Our Blog

Ongoing observations by End Point people

Random Strings and Integers That Actually Aren’t

By Josh Williams
July 2, 2020

rowntree’s randoms sweets

Image from Flickr user fsse8info

Recently the topic of generating random-looking coupon codes and other strings came up on internal chat. My go-to for something like that is always this solution based on Feistel networks, which I didn’t think was terribly obscure. But I was surprised when nobody else seemed to recognize it, so maybe it is. In any case here’s a little illustration of the thing in action.

Feistel networks are the mathematical basis of the ciphers behind DES and other encryption algorithms. I won’t go into details (because that would suggest I fully understand it, and there are bits where I’m hazy) but ultimately it’s a somewhat simple and very fast mechanism that’s fairly effective for our uses here.

For string generation we have two parts. For the first part we take an integer, say the sequentially generated id primary key field in the database, and run it through a function that turns it into some other random-looking integer. Our implementation of the function has an interesting property: If you take that random-looking integer and run it back through the same function, we get the original integer back out. In other words…

cipher(cipher(n)) == n

…for any integer value of n. That one-to-one mapping essentially guarantees that the random-looking output is actually unique across the integer space. In other words, we can be sure there will be no collisions once we get to the string-making part.

The original function is based off the code on the PostgreSQL wiki with just a few alterations for clarity, and should work for any modern (or archaic) version of Postgres.

CREATE OR REPLACE FUNCTION public.feistel_crypt(value integer)
  RETURNS integer
  LANGUAGE plpgsql
  IMMUTABLE STRICT
AS $function$
DECLARE
    key numeric;
    l1 int;
    l2 int;
    r1 int;
    r2 int;
    i int:=0;
BEGIN
    l1:= (VALUE >> 16) & 65535;
    r1:= VALUE & 65535;
    WHILE i < 3 LOOP
        -- key can be any function that returns numeric between 0 and 1
       ...

postgres python tips

Randomly spacing cron jobs

By Jon Jensen
June 30, 2020

bird footprints in snow

Cron is the default job scheduler for the Unix operating system family. It is old and well-used infrastructure — it was first released 45 years ago, in May 1975!

On Linux, macOS, and other Unix-like systems, you can see any cron jobs defined for your current user with:

crontab -l

If nothing is printed out, your user doesn’t have any cron jobs defined.

You can see the syntax for defining the recurring times that jobs should run with:

man 5 crontab

Important in that document is the explanation of the space-separated time and date fields:

field          allowed values
-----          --------------
minute         0-59
hour           0-23
day of month   1-31
month          1-12 (or names, see below)
day of week    0-7 (0 or 7 is Sunday, or use names)

A field may contain an asterisk (*), which always stands for "first-last".

For example, to make a job run every Monday at 3:33 am in the server’s defined time zone:

33 3 * * 1 /path/to/executable

Random interval scheduling

Sometimes it may be good to schedule a cron job to run at a somewhat random time: generally not truly random, but maybe at an arbitrary time within a specified time range rather than at a specific recurring interval.

This can be useful to keep simultaneous cron jobs for different users from causing predictable spikes in resource usage, or to run at a time other than the start of a new minute, since cron’s interval resolution doesn’t go smaller than one minute.

There isn’t any simple built-in way to randomize the scheduling in classic cron, but there are several ways to get it done:

cronie RANDOM_DELAY

The version of cron included with Red Hat Enterprise Linux (RHEL), CentOS, and Fedora Linux is cronie. It allows us to set the variable RANDOM_DELAY for this purpose. From its manual:

The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable. The random scaling factor is determined during the cron daemon startup so...


sysadmin automation

Improving max() performance in PostgreSQL: GROUP BY vs. CTE

By David Christensen
June 30, 2020

Spice Baazar Photo by Maxpax, used under CC BY-SA 2.0, cropped from original.

When working with large tables, even simple actions can have high costs to complete. What queries are acceptable for smaller tables can often be less than ideal when applied to large tables, so your specific choice of approach to a given problem becomes more important.

Note: We are using PostgreSQL 12, which supports some nice features like parallel btree index building, which can speed up parts of this process compared to earlier versions. We are using the default settings for this, which lets PostgreSQL use up to 2 parallel backend workers to speed up some operations.

Say you have a table table_a with multiple grouping fields field_a and field_b and you want to find the maximum value of another field field_c for each group.

The direct approach is to do something like the following:

SELECT field_a, field_b, max(field_c) FROM table_a GROUP BY 1,2;

This is functional and very straightforward. However, even if you have an index on (field_a, field_b, field_c), this can end up taking quite a long time if the tables are large. Let’s look at an actual example and the numbers we use.

First, let’s create our table:

CREATE TABLE table_a (field_a varchar, field_b integer, field_c date);

And populate it with some data:

INSERT INTO table_a
SELECT field_a, field_b, now()::date + (random() * 100)::int AS field_c
FROM unnest(array['AAA','BBB','CCC','DDD','EEE','FFF']) field_a,
    generate_series(1, 10000) field_b,
    generate_series(1, 1000);

This statement will populate this table with 60 million rows, consisting of 1000 random dates per each field_a, field_b pair; our task will now be to see how to efficiently find the max value for field_c for each grouping.

Let’s now create an index on all 3 fields:

CREATE INDEX ON table_a (field_a, field_b, field_c);

For the purposes of sanity/​clarity when testing approaches, let’s VACUUM and ANALYZE that table:

VACUUM ANALYZE table_a;

And let’s check...


postgres database

Job opening: PHP / JavaScript developer

By Jon Jensen
June 29, 2020

waterfall and mountains

We are looking for a PHP software engineer to work with us during business hours somewhere in the UTC-7 to UTC-4 time zones (U.S. Pacific to Eastern Time). This role can be full-time or part-time.

We are an Internet technology consulting company based in New York City, with 50 employees serving many clients ranging from small family businesses to large corporations. The company turns 25 years old this year!

Even before COVID-19 most of us worked remotely from home offices. We collaborate using SSH, GitHub, GitLab, chat, video conferencing, and of course email and phones.

What you will be doing:

  • Develop new web applications and support existing ones for our clients
  • Work together with End Point co-workers and our clients’ in-house staff
  • Use your desktop OS of choice: Linux, macOS, Windows
  • Use open source tools and contribute back as opportunity arises

What you bring:

Professional experience developing and supporting web applications in these technical areas:

  • 5+ years of development with PHP and front-end JavaScript
  • Frameworks such as Symfony, Laravel, Magento and Vue.js, React, Angular
  • Databases such as PostgreSQL, MySQL, MongoDB, Redis, Solr, Elasticsearch, etc.
  • Security consciousness
  • Git version control
  • Automated testing
  • HTTP, REST APIs
  • Bonus for familiarity with another ecosystem such as Ruby on Rails, Python/​Django, Java, .NET/​C#, Node.js, Go, Rust, Scala, Kotlin, Swift …

These work traits are just as important:

  • Strong verbal and written communication skills
  • An eye for detail
  • Tenacity in solving problems
  • A feeling of ownership of your projects
  • Work both independently and as part of a team
  • Focus on customer needs
  • A good remote work environment

What work here offers:

  • Collaborate with knowledgeable, friendly, helpful, and diligent co-workers around the world
  • Flexible, sane work hours
  • Annual bonus opportunity
  • Freedom from being tied to an office location
  • For full-time staff: paid holidays and vacation
  • For U.S. employees: health insurance subsidy...

company jobs php remote-work

Magento 2: Creating a custom theme

By Juan Pablo Ventoso
June 24, 2020

blue and yellow paint from a tube on a canvas Photo by Maria Eklind, CC BY-SA 2.0

In my previous post, we went through the steps needed to create a custom module in Magento 2. While modules consist of a set of classes to add new features to Magento, a theme controls how these features, and the entire website in general, will be displayed to the user. As stated in the Magento guide, a theme uses a combination of custom templates, layouts, styles, and images to provide a consistent look and feel across a Magento store.

Creating a new Magento 2 theme

We can create a theme based on a default “parent” theme or create a standalone theme from scratch. In most cases, I would recommend the first option. For this example, we will use Luma as our parent theme. The other option would be inheriting from the default “blank” theme.

Here’s an initial task list to get our new theme ready:

  • Create a new directory for the theme
  • Create the registration.php script
  • Create the theme.xml information file
  • Activate the new theme

Creating a new directory for the theme

While all our backend code should go in app/code, the frontend content is expected to go in app/design. And as our theme will only apply design changes to the frontend content, we should create the new directory for it under the path app/design/frontend. If we want to create a theme for the admin area instead, we need to create the directory inside app/design/adminhtml.

Let’s create a directory named EndPoint (our vendor name, continuing with the example from our previous article) and a subdirectory inside it, MyTheme:

cd {website_root}
mkdir -p app/design/frontend/EndPoint/MyTheme

Creating registration.php

Similar to the file we created for our module, registration.php tells Magento to register the new theme with the name and location we specify. Our file will be located at app/design/frontend/EndPoint/MyTheme/registration.php and should have the following content:

<?php
\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component...

magento php ecommerce

Linux Development in Windows 10 with Docker and WSL 2

By Kevin Campusano
June 18, 2020

Banner

I’m first and foremost a Windows guy. But for a few years now, moving away from working mostly with .NET and into a plethora of open source technologies has given me the opportunity to change platforms and run a Linux-based system as my daily driver. Ubuntu, which I honestly love for work, has been serving me well by supporting my development workflow with languages like PHP, JavaScript and Ruby. And with the help of the excellent Visual Studio Code editor, I’ve never looked back. There’s always been an inclination in the back of my mind though, to take some time and give Windows another shot.

With the latest improvements coming to the Windows Subsystem for Linux with its second version, the new and exciting Windows Terminal, and Docker support for running containers inside WSL2, I think the time is now.

In this post, we’ll walk through the steps I took to set up a PHP development environment in Windows, running in a Ubuntu Docker container running on WSL 2, and VS Code. Let’s go.

Note: You have to be on the latest version of Windows 10 Pro (Version 2004) in order to install WSL 2 by the usual methods. If not, you’d need to be part of the Windows Insider Program to have access to the software.

What’s new with WSL 2

This is best explained by the official documentation. However, being a WSL 1 veteran, I’ll mention a few improvements made which have sparked my interest in trying it again.

1. It’s faster and more compatible

WSL 2 introduces a complete architectural overhaul. Now, Windows ships with a full Linux Kernel which WSL 2 distributions run on. This results in greatly improved file system performance and much better compatibility with Linux programs. It’s no longer running a Linux look-alike, but actual Linux.

2. It’s better integrated with Windows

This is a small one: we can now use the Windows explorer to browse files within a WSL distribution. This is not a WSL 2 exclusive feature, it has been there for a while now. I think it’s worth mentioning...


windows linux docker php

Jamstack Conf Virtual 2020: Thoughts & Highlights

By Greg Davidson
June 16, 2020

Conference

Welcome to Jamstack Conf Virtual 2020

Last week I attended Jamstack Conf Virtual 2020. It had originally been slated to take place in London, UK but was later transformed into a virtual event in light of the COVID-19 pandemic. The conference began at 2pm London time (thankfully I double-checked this the night before!)—​6am for those of us in the Pacific Time Zone.

Before getting too much further I wanted to mention that if you are not familiar with the Jamstack, You can read more about it at jamstack.org.

To virtually participate in the conference we used an app called Hopin. I had not heard of it before but was impressed with how well it worked. There were over 3000 attendees from 130+ countries one of the times I checked. Phil Hawksworth was the Host/​MC for the event and did a great job. There were virtual spaces for the stage, sessions, expo (vendors), and networking. If you opted to, the networking feature paired you with a random attendee for a video chat. I’m not sure what I expected going into it but I thought it was fun. I met a fellow developer from the Dominican Republic. The experience was very similar though more serendipitous than the hallway track or lunch line at an in-person conference.

Phil Hawksworth welcoming the attendees

Keynote

Matt Biilmann opened the conference with a keynote address about the challenges we face as a developer community trying to improve access to accurate, timely and locally relevant information to a global audience. Many billions of users with all kinds of devices and varying levels of connectivity. He moved on to share how Netlify is trying to enable developers to “build websites instead of infrastructure” and “ensure all the best practices become common practices” through features like git-based deployments, build plugins, and edge handlers (more on those later).

State of the Jamstack Survey results

Laurie Voss reporting findings from the Jamstack Survey 2020

Laurie Voss walked us through the results of the...


jamstack html css javascript conference development

Why upgrading software libraries is imperative

By Selvakumar Arumugam
June 10, 2020

Image 0

Image by Tolu Olubode on Unsplash

Applications primarily run on front- and back-end programming languages, including library dependencies. Operating systems and programming languages can be periodically updated to run on the latest version, but what about the many libraries being used in the app’s front and backend? As we all know, it can be quite a daunting task to maintain and individually update a long list of software dependencies like the examples later in this post. Still, it is important to keep them updated.

This post dives into our experience upgrading a complex app with a full software stack and lots of dependencies. We’ll examine the benefits of upgrading, what you will need, and how to go about such an upgrade as simply as possible.

The app in question contained decade-old software and included extensive libraries when we received it from our client. The app used languages including Java, Scala, Kotlin, and JavaScript along with many libraries. The initial plan was to upgrade the complete software stack and libraries all at once due to the gap between versions. This proved to be more difficult than expected due to a host of deprecated and removed functionality as well as interdependence of a few of the libraries.

Conflict approach: “Don’t update unless you have to”

While this can be sustainable in the short term, it quickly becomes less applicable in the long run. One important purpose of updates is to (hopefully) protect from new vulnerabilities and cyber attacks. Scenarios arise where particular library fixes are implemented on the latest version, yet require upgrading other libraries to the latest version in a chain. Because upgraded libraries need extensive testing and preparation for new issues, this directly impacts whether the app attempts to resolve an issue.

Therefore, smaller and more frequent updates are more sustainable in the long run. Larger and less frequent upgrades will not only result in unexpected errors, but also require more...


software update
Page 1 of 188 • Next page

Popular Tags


Archive


Search our blog