Our Blog

Gears image by Guy Sie, CC BY-SA 2.0, cropped & scaled

systemctl: Let’s get back to basics

''Help me systemd, you are my only hope.''

Sometimes going back to day zero brings clarity to what seems like hopeless or frustrating situation for users from the Unix SysV init world. Caveat: I previously worked at Red Hat for many years before joining the excellent team at End Point and I have been using systemd for as long. I quite honestly have forgotten most of the SysV init days. Although at End Point we work daily on Debian, Ubuntu, CentOS, and BSD variants.

Here is a short and sweet primer to get your fingers wet, before we dive into some of the heavier subjects with systemd.

Did you know that systemd has many utilities you can run?

• systemctl
• timedatectl
• journalctl
• loginctl
• systemd-notify
• systemd-analyze - analyze system
• systemd-cgls - show cgroup tree
• systemd-cgtop
• systemd-nspawn

And systemd consists of several daemons:

• systemd
• journald
• networkd
• logind
• timedated
• udevd
• system-boot
• tmpfiles
• session

That’s a long way from the old SysV init days. But in all essence it’s not that different. The one thing that stands out to me is we have more information with less typing then previously. That can only be a good thing, right?

Well, let’s see! There are many many web pages out there that list systemd or systemctl switches/​flags. However in everyday use I want to speed up the work I do, I want information at my fingertips, and I find flags and switches which mean something sure do make it easier.

Pro Tip 1: Tab completion

Before you begin playing with the commands, you should install bash-completion. Some distros don’t auto-complete with systemd until you install that, and without tab auto-completion you miss out on a lot of systemctl.

As an example when you tab for completion you will see many of the systemctl options:

# systemctl
add-requires           enable                 is-system-running      preset                 show
add-wants              exit ...

TLS shutdowns are real

The payment gateways have been warning for years about the impending and required TLS updates. Authorize.net and PayPal—to name a few—have stopped accepting transaction requests from servers using TLS 1.0. Despite the many warnings about this (and many delays in the final enforcement date), some projects are affected by this and payments are coming to a stop, customers cannot checkout, and e-commerce is at a standstill.

Ideally, getting to security compliance would include a larger migration to update your underlying operating system and your application. But a migration and software update can be an expensive project and in some cases, the business can’t wait weeks while this is done.

End Point has worked with several clients recently to try to remedy the situation by using a reverse proxy to fix this and we’ve had good success on getting payments flowing again.

What is a proxy?

A proxy is a mid-point, essentially a digital middleman, moving your data from one place to another. In two recent client instances, we ended up using nginx (the stack’s webserver) as the reverse proxy, basically running a separate server for just shuttling requests to/​from the payment gateway. Since we want to be able to run the gateway in both live and test modes, we use two separate server definitions in our nginx include, one for each.

Since the proxy is talking to the gateway in TLS 1.2 the payment gateway is happy. Since the application can talk http to the proxy running on the same machine, your application is happy. Since payments are now flowing, the business is happy.

Why use a proxy?

While we always impress on clients the importance of staying up-to-date with their entire stack (operating system, language, application frameworks), this is not always practical for some sites, whether for cost reasons or some technical limitations which keep them on a specific library or framework version. In our case, these clients had been migrated to CentOS 7...

The first time I got paid for doing PostgreSQL work on the side, I spent most of the proceeds on the mortgage (boring, I know), but I did get myself one little treat: a boxed set of DVDs from a favorite old television show. They became part of my evening ritual, watching an episode while cleaning the kitchen before bed. The show features three military draftees, one of whom, Frank, is universally disliked. In one episode, we learn that Frank has been unexpectedly transferred away, leaving his two roommates the unenviable responsibility of collecting Frank’s belongings and sending them to his new assignment. After some grumbling, they settle into the job, and one of them picks a pair of shorts off the clothesline, saying, “One pair of shorts, perfect condition: mine,” and he throws the shorts onto his own bed. Picking up another pair, he says, “One pair of shorts. Holes, buttons missing: Frank’s.”

The other starts on the socks: “One pair of socks, perfect condition: mine. One pair socks, holes: Frank’s. You know, this is going to be a lot easier than I thought.”

“A matter of having a system,” responds the first.

I find most things go better when I have a system, as a recent query writing task made clear. It involved data from the Instituto Nacional de Estadística y Geografía, or INEGI, an organization of the Mexican government tasked with collecting and managing country-wide statistics and geographical information. The data set contained the geographic outline of each city block in Mexico City, along with demographic and statistical data for each block: total population, a numeric score representing average educational level, how much of the block had sidewalks and landscaping, whether the homes had access to the municipal sewer and water systems, etc. We wanted to display the data on a Liquid Galaxy in some meaningful way, so I loaded it all in a PostGIS database and built a simple visualization showing each city block as a polygon extruded from the earth, with...

Earthquakes and explosive eruptions are currently rocking Kīlauea’s summit crater, creating concerns for the local community. Fortunately, NOAA’s Mokupāpapa Discovery Center in Hilo is stepping up to educate the greater community with the help of their Liquid Galaxy system, which was created and is supported by End Point.

While there is active volcanic activity and explosive eruptions continue at the Kīlauea summit, Hawaiʻi Volcanoes National Park is mostly closed to its nearly two million annual visitors. NOAA’s Mokupāpapa Discovery Center in Hilo is helping to support the community affected by the lava flows and eruption at Kīlauea summit and along its Lower East Rift Zone by hosting Hawaiʻi Volcanoes National Park Service rangers and interpretive staff.

To lessen the impact on park visitors and to provide a venue to learn about the current eruption, NOAA’s Mokupāpapa Discovery Center in Hilo is hosting a pop-up park center, with daily ranger talks at 10 am and 2 pm, on-site rangers throughout the day, and support of park programming. NOAA National Weather Service meteorologists from the Hilo Data Collection Office are also participating in the daily 10 am briefing to provide information on ash fall, wind direction, and air quality hazards.

The briefings are being given using End Point’s Liquid Galaxy as a visualization and briefing tool to show where the current lava flows are and where ash fall may occur from the explosive eruptions at the summit. Understanding where the activity is taking place, as well as understanding what areas are potentially dangerous, has been critical to keeping the public safe from this spectacular natural event. Liquid Galaxy is proving to be an excellent tool to show both the geography and the geology, and previous historic flows on Hawaiʻi Island.

Our End Point support team is monitoring this Liquid Galaxy system 24×7 to ensure there are no disruptions in service for the public’s education. Although the need for our system...

Photo by Julio Albarrán, CC BY-SA 2.0, cropped

The European Union’s General Data Protection Regulation (GDPR) that became law a little over two years ago, is now implemented as of 25 May 2018.

Another GDPR article?

Over the past few weeks most of us have been receiving lots of GDPR-related email from companies sending us new privacy policies, so most people have heard at least something about GDPR. But we are finding that some still do not know the impact on their business, and they wonder if it has anything to do with them if they are outside the EU. This article is our attempt to help set those people on the path to finding answers.

I think the first thing to recognize is that the GDPR is a general business matter, not primarily a technical matter. The regulation focuses on business processes and information management (whether computerized or not), and law; it is not actually about software or legal verbiage on websites.

The GDPR is not the kind of law that can be complied with simply by adding a few features to software, changing a few configuration options, or updating a legal notice and moving on with no changes to actual practice.

Some people outside the EU wonder how it is possible that this regulation can affect them. Consider as an example that United States income tax law affects US citizens anywhere in the world, US banking rules affect banks in most parts of the world, and US sanctions affect people and businesses anywhere the world. The EU has a long reach as well.

As always, you may want to discuss questions about all this with a competent lawyer for clarification, to go into specifics about your business situation and get tailored advice.

Who does the GDPR apply to?

The GDPR applies if (citing an EU guidance document):

• your company processes personal data and is based in the EU, regardless of where the actual data processing takes place; or
• your company is established outside the EU but offers goods or services to, or monitors the behavior...

Photograph by Helena Lopes, CC0

I recently had the chance to spend my weekend enhancing my knowledge by joining a local community meetup in Malaysia which is sponsored by Malaysian Global Innovation & Creativity Centre (MaGIC). The trainer was Mr Lee Boon Kong.

Anaconda and Jupyter Notebook

We started by preparing our Jupyter Notebook setup which is running on the Anaconda Python distribution. The installer is 500 MB in size but pretty handy when we started using it.

Anaconda comes with a graphical installer called “Navigator” so the user can install some packages for work. However it did not always work for me on some OSes, so I had to use its command-line based tool “conda”. Conda works like Linux-based package management tools such as apt, dnf, yum, and pacman, so to install a package I would just run conda install <package name>.

Jupyter uses a web browser to allow us to write the code directly in its cell. It is quite helpful for us to debug the code or if we just want to execute it segment by segment independently.

Creating Twitter’s API key

First we need to head to apps.twitter.com.

The following items are needed:

• Consumer Key (API key)
• Consumer Secret (API secret)
• Access Token
• Access Token Secret

Using Tweepy, NLTK and TextBlob

from textblob import TextBlob
import tweepy
import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
consumer_token = '<put your token here>'
consumer_secret = '<put your secret here>'
access_token = '<put your access token here>'
access_secret = '<put your access secret here>'
auth = tweepy.OAuthHandler(consumer_token, consumer_secret)
auth.set_access_token(access_token, access_secret)
api = tweepy.API(auth)
public_tweets = api.search("Avengers Infinity War", lang='en')
print("number of tweets extracted: " + str(len(public_tweets)))
for tweet in public_tweets:
    print(tweet.text)
    analysis = TextBlob(tweet.text)
    print(analysis.sentiment)
    print("\n")

If we want to increase the number...

Shared culture

Having some shared culture is important for working together well, and we can build that culture on familiar terminology, understanding, experiences, stories, and ideas.

To help give all of us at End Point some common reference points, we have collected a set of valuable articles and books that we encourage everyone to read.

Some of these have been standard reading at End Point for more than a decade, while others have been added over the years since we began doing this. Some are short and simple, others more in-depth. Our list is intentionally general, mostly avoiding specific technologies that only a subset of us use.

No one article or book can be entirely authoritative for all situations, but each gives us more of the wisdom out there to consider and judiciously apply where it fits.

Slow and steady

When new employees start at End Point, we ask them to read the articles during their first week or so, and the relevant books within roughly their first year.

Reading only a little at each sitting and spreading the reading out over time allows the ideas to sink in gradually and be incorporated into our work, rather than overwhelming with new information that cannot all be absorbed at once.

For everyone

Because we work in the software development industry, it is important that not only technical people such as developers, database experts, and system administrators be part of the shared culture. Everyone else at End Point including designers, project managers, sales, marketing, etc. should also be familiar with these articles and the terms and concepts they discuss:

• Speed matters: Why working quickly is more important than it seems by James Somers
• Please don’t just say hello in chat by Brandon High
• Learn to Read the Source, Luke by Jeff Atwood—​practical reasons free software and open source matter, and why actually using the source code matters
• What is free software? by the Free Software Foundation and Richard Stallman
• The Open Source Definition

We had the unique opportunity to participate in the EarthX conference held this week in Dallas, Texas. EarthX brings over 100,000 visitors together to hear about environmentalism, recycling, ocean preservation, sustainability tech advances, and how to be good custodians of our Earth.

We set up 3 systems at the conference to give the widest coverage for the wonderful content developed by our partners:

• NOAA showcased new 360 photos from Palmyra Atoll, a tiny dot of land 1000 miles due south of Hawaii in the middle of a National Marine Reserve, along with an incredible 360° video from Christophe Baillache and Sophie Ansel.
• Blue Abyss showcased their new 50-meter deep diving tank soon to be installed in the UK, where they will support training for space travel or run oceanic simulations under very controlled conditions.
• We also showed “Diving with Sylvia”, a VR game built by Cascade Game Foundry for the Oculus Rift. This was tricky, as it involved simultaneous visualizations to both the VR goggles wearer as well as the 7 screens of the Liquid Galaxy.

While at the conference, we attended a wonderful gala with Sylvia Earle, one of the premiere environmental scientists who focuses specifically on oceanic wildlife and how to preserve what she calls “Hope Spots” around the world where biodiversity is especially rich and especially at risk. (Yes, we have a presentation that flies to all these “blue hope” spots.)

Sylvia and other speakers emphasized the need for telling the story of these places, as a good story is key to raising awareness, building consensus around preservation, and leading the way for policy change. This same need of telling a strong environmental message was echoed by our friends at NOAA, the National Marine Sanctuary Organization, and even the President of Palau, a Pacific Island nation who has taken the lead and now preserves 80% of its waters from fishing or exploitation!

Our 3 Liquid Galaxy systems at the conference performed admirably and...