October 2, 2018
We are looking for a full-time, salaried engineer to help us further develop our software, infrastructure, and hardware integration for our shared immersive system, End Point Liquid Galaxy.
End Point is a technology consulting company founded in 1995 and based in New York City, with around 50 employees working from our offices in New York City, the tri-cities area in eastern Tennessee, and from home offices around the globe. We serve over...
September 12, 2018
In a past blog post I described a method I’d used to digest raw statistics from the Mexican government statistics body, INEGI, quantifying the relative educational level of residents in Mexico City. In the post, I divided these data into squares geographically, and created a KML visualization consisting of polygons, where each polygon’s color and height reflected the educational level of residents in the corresponding area. Where the original data proved slow to render and difficult to interpret, the result after processing was visually appealing, intuitively meaningful, and considerably more performant. Plus I got to revisit a favorite TV show, examine SQL’s Common Table Expressions, and demonstrate their use in building complex queries.
But the post left a few loose ends untied. For instance, the blog post built the visualization using just one large query. Although its CTE-based design rendered it fairly readable, the query remained far too complex, and far too slow, at least for general use. Doing everything in one query makes for a sometimes enjoyable mental exercise, but it also means the query has to start from zero, every time it runs, so iterative development bogs down quickly as the query must recalculate at every new iteration all the expensive stuff it already got worked out in previous runs. Further, the query in that post calculated its grid coordinates all on its own; there’s an easier method I promised to introduce, which I’d like to demonstrate here.
GIS databases contain information of (at least) two different types: vector data, and raster data. Vector data describes points, connected with lines and curves, and various objects derived from them, such as polygons and linestrings. Rasters, on the other hand, represent a two dimensional array of pixels, where each pixel corresponds to a certain geographic area. Each pixel in the raster will have two sets of associated coordinates: first, latitude and longitude values describing the geographic...
August 30, 2018
If you like to solve business problems and can take responsibility for getting a job done well without intensive oversight, please read on!
End Point is a 23-year-old web consulting company based in New York City, with about 50 employees, the majority working remotely from home offices. We collaborate using SSH and Remote Desktop, GitLab, GitHub, chat, Hangouts and other conferencing software, and good old email and phones.
We serve over 200 clients ranging from small family businesses to large corporations.
We are experts in web development, databases, and DevOps, and we use mostly open source frameworks, in a variety of languages, on Linux and Windows.
August 29, 2018
The field of Reinforcement Learning has seen a lot of great improvement in the past years. Researchers at universities and companies like Deep Mind have been developing new and better ways to train intelligent, artificial agents to solve more and more difficult tasks. The algorithms being developed are requiring less time to train. They also are making the training much more stable.
This article is about an algorithm that’s one of the most cited lately: A3C — Asynchronous Advantage Actor-Critic.
As the subject is both wide and deep, I’m assuming the reader has the relevant background mastered already. Although reading it might be interesting even without understanding most of the notions in use, having a good grasp of them will help you get the most out of it.
Because we’re looking at the Deep Reinforcement Learning, the obvious requirement is to be acquainted with the neural networks. I’m also using different notions known in the field of Reinforcement Learning overall like $Q(a, s)$ and $V(s)$ functions or the n-step return. The mathematical expressions, in particular, are given assuming that the reader already knows what the symbols stand for. Some notions known from other families of RL algorithms are being touched on as well (e.g. experience replay) — to contrast them with the A3C way of solving the same kind of problems. The article along with the source code uses the OpenAI gym, Python, and PyTorch among other Python-related libraries.
The A3C algorithm is a part of the greater class of RL algorithms called Policy Gradients.
In this approach, we’re creating a model that approximates the action-choosing policy itself.
Let’s contrast it with value iteration, the goal of which is to learn the value function and have policy emerge as the function that chooses an action transitioning to the state of the greatest value.
August 6, 2018
Over the years, Liquid Galaxy has taken us to many a far-flung locale, but few of us at End Point were expecting a Liquid Galaxy installation in Africa. Yet that is exactly what happened: In conjunction with Google’s “Next Billion Users” initiative, End Point was invited to install a Liquid Galaxy at “Google for Nigeria” in Lagos, Africa’s largest city, a teeming metropolis of 21 million souls.
With the challenge of getting our equipment to Nigeria and hearsay about intermittent Internet and electricity, we had more than our usual concern about making sure our Liquid Galaxy would function reliably for the event. And given the US Department of State’s warnings about health and safety, the trip would be more involved than most. However, End Point engineer Will Plaut fairly leapt at the chance. More than 20 hours of flying, a bevy of vaccines and a host of visa procedures were no match for Will’s intrepid spirit and sense of adventure — he even took it upon himself to hunt down the difficult-to-find Yellow Fever vaccine in Greenville, South Carolina.
But before simply sending Will off to fend for himself amidst the hubbub of Lagos, the list of logistics problems seemed unending. The first order of business was to find a local contact to advise us both on the vagaries of shipping and customs as well as to help our engineer set the system up, keep him out of harm’s way, and give him a real taste of Nigeria. A tall order. Luckily, End Point had a personal contact who put us in touch with Olufunmilola “Afeez” Atiku, fixer extraordinaire. Afeez was not only able to explain to us the protocol for shipping (the LG could not be simply shipped to the airport in Lagos, then picked up, but had to be entrusted to a local shipper), he also arranged for Will to be picked up at the airport, whisked through customs, and accompanied & assisted 24×7 for the next 6 days by the joyous and trusted Taofik. The driving chores were delegated to an aptly named fellow known only as “Happiness...
July 27, 2018
As the share of internet traffic passing through mobile devices hovers around 50%, it’s easy to forget that a lot of computing still happens on desktops and laptops—and perhaps even easier to forget that the majority of those desktops and laptops are running Microsoft Windows.
At End Point, we take pride in being a multi-platform organization and using open source technologies in real-world business environments. Our internal chat service, wiki, and many other libraries and tools we employ across our client base are all open source. However, for various reasons many companies choose Microsoft’s solutions including long-time standards Windows Server, Active Directory, and Exchange.
With the acquisition of Series Digital in 2017 and the addition of Dan Briones to our staff, End Point has begun supporting Windows infrastructure in a new way. In 2018 we’ve recently added two new engineers to the mix, Chris Hopkins and Charles Chang, to form a full-service Windows consulting team, all local to our New York City office on Park Avenue.
Dan is the team lead and has over three decades of hands-on experience in IT systems management, systems integrations, migrations, virtualization, networking, security, orchestration, security, compliance, and maintenance, specializing in the Microsoft Windows ecosystem. He also develops applications using the .NET framework and SQL Server database. Dan works with many of our local New York clients, focusing on their network and infrastructure.
Charles has 21 years of experience in IT ranging from managing Windows infrastructure, to datacenter relocation and hardware refresh initiatives, office relocation, virtualization, disaster recovery and business continuity strategy. Charles has worked extensively with major vendors such as VMware, IBM, EMC, Cisco, and Dell in building enterprise environments in the datacenter. He also has experience in implementing data loss prevention technology, web and email filtering...
July 17, 2018
We all like how apps like Spotify or Last.fm can recommend us a song that feels so much like our taste. Being able to recommend an item to a user is very important for keeping and expanding the user base.
In this article I’ll present an overview of building a recommendation system. The approach here is quite basic. It’s grounded though in a valid and battle-tested theory. I’ll show you how to put this theory into practice by coding it in Python with the help of MXNet.
The general setup of the content recommendation challenge is that we have users and items. The task is to recommend items to a particular user.
There are two distinct approaches to recommending content:
The first one bases its outputs on the the intricate features of the item and how they relate to the user itself. The latter one uses the information about the way other, similar users rank the items. More elaborate systems base their work on both. Such systems are called hybrid recommender systems.
This article is going to focus on collaborative filtering only.
In the simplest terms, we can represent interactions between users and items with a matrix:
In the above case users can rate items on the scale of
<-1, 1>. Notice that in reality it’s most likely that users will not rate everything. The missing ratings are represented with the dash:
Just by looking at the above table, we know that no amount of math is going to change the fact that user1 completely dislikes item1. The same goes for user2 liking item2 a lot. The ratings we already have make up for a fairly easy set of items to propose. The goal of a recommender is not to propose the items users know already though. We want to predict which of the “dashes” from the table are most likely to be liked the most. Putting it in...
July 14, 2018
Several of our clients run ecommerce sites, built on custom software, that allow customers to pay in their choice of a few different currencies.
Some have set up separate merchant bank accounts for each currency and use separate payment gateway accounts to accept payments natively in each currency. But more commonly they use a payment gateway that allows them to accept payment in several currencies, but receive funds converted into their specified native currency in their merchant bank account.
In either case, it is common to store prices for products, shipping, etc. in one base currency (say, USD for companies based in the U.S.) and dynamically convert prices for customers. Non-native prices may need to be marked up to cover the cost of conversion into the native currency, depending on the terms of the agreement with the payment gateway or bank.
Because currency exchange rates change often, and because payment gateways generally do not offer a way to retrieve the exchange rates in advance, we need our own source for ongoing retrieval and storage of our exchange rates (also known as forex rates).
For a while we were very pleased with Fixer.io, which was a free service that collected exchange rates from the European Central Bank (ECB) and provided current or historical rates via a simple JSON API. We were sad to find that in March 2018 they deprecated that API and in June 2018 they discontinued it entirely, as described to their users. Fixer.io has transitioned to a paid service and they appear to have improved their operation to retrieve exchange rate data from at least a dozen more sources than the ECB, and to store far more frequent rate updates. Those are nice features, but not something our clients need.
Fixer.io still offers a free plan that allows 1000 API calls per month but in that plan the exchange rates are all based on the Euro. You need to sign up for at least the $10/month plan to retrieve exchange rates from the other currencies as base. As our...