Our Blog

Ongoing observations by End Point people

A Tool to Compare PostgreSQL Database Schema Versions

By Selvakumar Arumugam
February 11, 2020

Parcel sorting Photo by @kelvyn on Unsplash

The End Point development team has completed a major application migration from one stack to another. Many years ago, the vendor maintaining the old stack abandoned support and development. This led to a stack evolution riddled with independent custom changes and new features in the following years.

The new application was developed by a consortium that created migration scripts to transfer data to a fresh stack resulting in a completely restructured database schema. While we could not directly use those consortium migration scripts to our client application, attempting to create migration scripts from scratch would be tedious due to the many labor-intensive and time-consuming tasks. We looked to reuse and customize the scripts in order to ensure an exact match of the customized changes to the client’s application.

Liquibase: A Schema Comparison Tool

After an arduous hunt for a suitable solution, we came across Liquibase, an open-source schema comparison tool that utilizes the diff command to assess missing, changed, and unexpected objects.

Installation and Usage

Let’s see how to use Liquibase and review the insights and results offered by the diff command.

Before beginning, download the latest version of Liquibase. As the default package doesn’t have its own driver, it would be wise to add the PostgreSQL driver to the Liquibase lib folder. (You’ll need to do this with any other database types and their necessary libraries and drivers.)

$ wget https://github.com/liquibase/liquibase/releases/download/v3.8.5/liquibase-3.8.5.tar.gz
$ tar xvzf liquibase-3.8.5.tar.gz
$ wget https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.5/postgresql-42.2.5.jar -P lib/

$ ./liquibase \
--classpath="/path/to/home/apps/liquidiff/lib" \
--outputFile=liquibase_output.txt \
--driver=org.postgresql.Driver \
--url=jdbc:postgresql://localhost:5432/schema_two \
--username=postgres \
--password=CHANGEME \
--defaultSchemaName=public \
Diff \
--referenceUrl...

postgres database

A primer on Java

By Lajos Árpád
February 10, 2020

What is Java and why is it interesting?

Java is a descendant of C++ and it is a C-based language. C was therefore the original language and it is probably not an overstatement to say that C is the most popular programming language in history. Most programmers speak at least C or one of its descendants. Let’s take a quick look at the most popular on the list of C-based languages:

  • C
  • C++
  • C#
  • Fantom
  • Go
  • Java
  • JavaScript
  • Objective C
  • Perl
  • PHP
  • Swift

Java is a member of a large family of programming languages and as a result, if someone learns Java, then they will have an easier time learning one of its cousins. And at the same time, if someone already speaks a C-based language, then Java is not too difficult to learn. Also, if someone is already a programmer and does not speak a C-based language yet, then it is in his/​her interest in most cases to learn a C-based language and thus to have an understanding of the most popular language family.

Let’s see the list of popular languages according to Stackify:

Yes, Java has lost almost a percent from its popularity according to the list above, but even though the trend is suboptimal for Java at the time of this writing, it’s still ahead of the second-placed C in a convincing manner, since between the two there is almost a whole percent and has an advantage of more than 6% over Python, which is not a C-based language, but nevertheless it’s popular and rapidly gaining popularity, but it’s still very far from Java. Any other language is way below the popularity of Java, not even close.

What this means:

  • If you get a programming job and you are agnostic to languages, then Java is a language you may well work in.
  • Due to the popularity of the language for most problems you will likely find a well written and well tested, reliable library.
  • If you search for a programmer, he/​she will more likely speak Java than other languages.

According to Towards Data Science, based on the data acquired from TIOBE, Java has been the most...


java wildfly language

End Point Security Tips: Securing your Infrastructure

By Charles Chang
February 5, 2020

phishingemail

Photo from comparitech.com

Implement Security Measures to Protect Your Organization & Employees

In this post, I’ll address what I believe are the three important initiatives every organization should implement to protect your organization and employees:

  1. Train employees on security culture.
  2. Implement the best technical tools to aid with organizational security.
  3. Implement recovery tools in case you need to recover from a security breach.

Habits of a Security Culture

Train everyone in your organization on these fundamentals:

  1. The only time you should be requested to reset your password by email is when you initiate it. There are rare exceptions to this rule, such as when accounts are compromised and providers request all users reset their passwords, but those events should be publicly announced. Staff can confirm with security personnel before acting on such requests.
  2. If you are asked to reset your password, it will typically be after you successfully logged into a website and the old one has expired.
  3. If you receive an email and do not know the sender, do not trust the contents or open attachments. Get advice from security personnel if needed.
  4. If you think the email is from your bank, keep in mind that banks do not ask their clients for private information via email.
  5. If you think the social security office emailed you to obtain your personal information, keep in mind that they do not initiate or solicit private information via email.
  6. Companies should not solicit private information unless you initiate first.
  7. Online retailers should not ask for your private information unless you initiate first.

A Security Concern: Going Phishing!

One of the more common ways to steal someone’s private information is through phishing. Phishing is like fishing: you try to catch something. In this case, the ‘fish’ is your data. Someone with malicious intent sends you email to attempt to get you to click on the link, picture, content, etc. within the...


security windows infrastructure

Ecommerce sales tax primer

By Elizabeth Garrett Christensen
January 13, 2020

Co-authored by Greg Hanson

Roman tax burning Source image

Tax collection is one of the topics du jour for those of us in the ecommerce industry. Since state and local authorities are now able to levy taxes on ecommerce goods, taxation for online stores has become quite complicated. The purpose of this post is to give you some next steps and ideas on implementation if you’re new to the topic and need to know how to get started on tax collection for your ecommerce business.

Current ecommerce sales tax policy stems from the 2018 U.S. Supreme Court decision South Dakota v. Wayfair, Inc. Since that decision, favoring South Dakota, 30 states have enacted legislation to require ecommerce stores to pay sales tax if they fit the definition of having an ‘economic nexus’, that is, they do enough business in the state to be worth taxing.

Talk to your Tax Accountant

So the first and most important note is to get your own legal counsel in regards to your taxes. There are many rules and things are changing every month with local and state authorities, so you’ll need reliable counsel on the topic.

If you’re looking for someone to help, make sure this person has:

  1. Knowledge about product variants. For example, clothing may not be taxed in some areas.
  2. Familiarity with tax policy in the entire country, and not just one local retail area.
  3. The ability to help you determine in which states you have a tax ‘nexus’ and need to pay sales tax. For many small businesses, you might only do enough business to pay sales tax in your home state and a few large ones like California or New York.

Research Software

Luckily for anyone starting to collect sales tax, there are some very good SaaS (software as a service) systems out there to make tax collection, reporting, and compliance easy. These software systems typically integrate with your ecommerce store by providing the store with the correct tax amount for the sale and collecting information for you on your reports and filing information to get the taxes...


ecommerce saas payments magento interchange

Decreasing your website load time

By Juan Pablo Ventoso
January 7, 2020

Decreasing our website load time Photo by Johan Larsson, used under CC BY 2.0

We live in a competitive world, and the web is no different. Improving latency issues is crucial to any Search Engine Optimization (SEO) strategy, increasing the website’s ranking and organic traffic (visitors from search engines) as a result.

There are many factors that can lead to a faster response time, including optimization of your hosting plan, server proximity to your main traffic source, or utilization of a Content Distribution Network (CDN) if you are expecting visitors on an international level. Some of these solutions and many others can be implemented with only a couple hours of coding.

Inline styles and scripts for the topmost content

Nobody enjoys waiting for long load times. When opening a Google search link, being met with a blank page or a loading GIF for several seconds can seem agonizing. That’s why optimizing the initial rendering of your page is crucial.

The content that immediately appears to the user without the need to scroll down is referred to as “above-the-fold”. This is where your optimization efforts should be aimed. So here’s a plan to load and display as quickly as possible:

  • First, differentiate the critical styles and scripts you need to render the topmost content, and separate them from the rest of our stylesheet and external script references.

  • Then, minify the separated styles and scripts, and insert them directly on our page template, right before the closing </head> tag.

  • Finally, take the stylesheet and scripts link references from the <head> tag (where it’s usually located) and move them to the end of the above-the-fold content.

Now, the user won’t have to wait until all references are loaded before seeing content. Tip: Remember to use the async tag on scripts whenever possible.

  • example.html:
<head>
    <style>{above-the-fold minified inline styles goes here}</style>
    <script type="text/javascript">{above-the-fold critical scripts goes here}</script>
</head>
<body>
...

seo optimization html csharp dotnet

Useful terminal tools

By Jon Jensen
January 3, 2020

Móricz Zsigmond körtér Underground Station (people, escalators)
Photo by Tee Cee · CC BY 2.0, cropped

Like most of my co-workers, I spend a lot of time in a terminal emulator (console) in a shell at the Linux command line. I often come across tools that make work there nicer, but sometimes I forget about them before I integrate them into my workflow. So here are notes about a few of them for myself and anyone else who may find them useful.

HTTPie

HTTPie is:

a command line HTTP client with an intuitive UI, JSON support, syntax highlighting, wget-like downloads, plugins, and more.

Given how commonly-used curl, wget, and GET/POST (lwp-request) are, it is nice to see some innovation in this space to enhance usability.

Here is a simple example that demonstrates several HTTP redirects with full request and response headers, colorized:

http -v --pretty=all --follow endpoint.com | less -R

The color highlighting of the body, not just response headers, is the main difference here from curl, wget, etc.

Also nice for ad-hoc interactive use is that the verbose header output is sent to stdout instead of stderr, so it shows up in less without needing to have the shell merge it with 2>&1 before piping to less.

An aside on HTTP redirects

In the above example, the client makes 3 requests, because the first 2 are redirects:

  • http://endpoint.com/
  • https://endpoint.com/
  • https://www.endpoint.com/

Normally we would want to reduce the number of HTTP redirects, so why not redirect straight from http://endpoint.com/ to https://www.endpoint.com/?

Before the introduction of HTTP Strict Transport Security (HSTS) to the web, that is what we did.

But with HSTS it is better to pass through HTTPS for each hostname, so that the Strict-Transport-Security HTTP response header can be sent and the browser can cache the fact that both the bare yourdomain.tld and www.yourdomain.tld should only be accessed via HTTPS.

See the thorough description at the Sentinel Stand blog post for more details, including a discussion of includeSubDomains traps.


tips tools

Making sense of XML/JSON items in the shell

By Muhammad Najmi bin Ahmad Zabidi
December 31, 2019

a shell

Working as a system administrator means I have to spend quite some time during my work (and even during casual surfing) with the terminal. Sometimes I feel that certain information I want could just be fetched and parsed through the terminal, without having to use my mouse and point to the browser.

Some of the websites I visit use XML and JSON, which we could parse with Bash scripting. Previously I wrote a Ruby script to call Nokogiri to parse the XML elements until I found a Bash tool that could do the same thing.

These tools have already been around for quite a while—I’d just like to share what I did with them. The tools I used are xmlstarlet for XML parsing and jq for JSON.

XML

I have the following XML elements, and I’ll save them to a file called data.xml:

<rss version="2.0"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
    xmlns:admin="http://webns.net/mvcb/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:content="http://purl.org/rss/1.0/modules/content/">

    <channel>
        <title>eSolat JAKIM : Waktu Solat Hari Ini</title>
        <link>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</link>
        <description>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</description>
        <dc:language>ms</dc:language>
        <dc:creator>www.e-solat.gov.my</dc:creator>
        <dc:rights>Copyright JAKIM</dc:rights>
        <dc:date>26-12-2019 00:37:31</dc:date>
        <admin:generatorAgent rdf:resource="expressionengine" />

        <item>
            <title>Imsak</title>
            <description>05:53:00</description>
        </item>
        <item>
            <title>Subuh</title>
            <description>06:03:00</description>
        </item>
        <item>
            <title>Syuruk</title>
            <description>07:14:00</description>
        </item>
        <item>
            <title>Zohor</title>
            <description>13:16:00</description...

shell json

A Career Talk for 1st Graders

By Steph Skardal
December 6, 2019

Steph giving a career talk Giving a Career Talk to 1st Graders

This week, I gave a career talk to my daughter’s 1st grade class and I talked about my job as a software engineer. I started with this video, which depicts two kids explaining to their dad how to make a peanut butter sandwich (called the “Exact Instructions Challenge”), but he takes them very literally and acts as though he has no context on how to work with peanut butter, jelly, and bread. The video got some giggles!

After the video, I talked about how the video was similar to what I do: I give computers instructions, and like that silly dad, computers don’t know anything about what they are being told to do. I hope they understood the analogy!

I have 5 Alexas!

We talked about what a computer is and how we all have a lot of computers at our house (“I have 5 Alexas at my house!”, “I have a PS4!”, “I have a PS2!”), even some that can turn the lights on and off now. I didn’t show them code because I didn’t think it would mean much to them, but a couple of the kids in the class had worked on kid-friendly coding projects.

I talked a little bit about my education (1st graders aren’t quite sure what this “college” thing is), and how I work from home. We talked about how I problem-solve, just like they do — right before the talk started, their teacher asked them to problem-solve so that everyone could have a chair! I explained that sometimes I problem-solve and figure out what the fastest way to do something is, or what another solution might be, but it might last longer. We also talked about how communication (reading and writing) are important in my job, so they should keep working on that!

Q&A

The best part of the talk was Q&A, because I was able to understand what was in their big kid brains! Here are some of the questions I fielded:

  • Do you give computers instructions in 1s and 0s? How do you do that? How do computers understand 1s and 0s? I tried to explain this a little bit by talking about different languages (a couple...

development community
Page 1 of 184 • Next page

Popular Tags


Archive


Search our blog