Safely Modifying Your Hosts File with Gas Mask

Sometimes the DNS for a specific domain on your machine needs to point somewhere else – at Curalate, we test changes microservices locally before shipping them, which could require redirecting requests to look at that local instance. One way to do this is by adding an entry like 127.0.0.1 some.service.curalate.com to /etc/hosts.

How to Setup a Scheduled Scala Spark Job

Have you written a Scala Spark job that processes a massive amount of data on an intimidating amount of RAM and you want to run it daily/weekly/monthly on a schedule on AWS? I had to do this recently, and couldn’t find a good tutorial on the full process to get the spark job running. Included in this article and accompanying repository is everything you need to get your Scala Spark job running on AWS Data Pipeline and EMR.

Hey, you busy? I have thousands of questions to ask you.

If you’re a brick-and-mortar business owner, you quickly identify patterns in your customer foot traffic, especially around the holidays when achieving your sales goals depends on both timely and quality service. If you’re an e-commerce business owner, like many of Curalate’s 1,000+ customers, it’s really no different: holiday sales are crucial to success and they depend heavily on your site’s reliability.

Uploading EC2 Logs to S3 on Shutdown

If you’ve ever used an auto scaling group (ASG) on AWS, you’ve probably had an EC2 instance fail and get removed from the ASG. While great for redundancy (the ASG launches a new instance to start handling requests), it makes debugging the failure difficult since the ASG terminates the bad instance, erasing any evidence of what went wrong. Below, I present a script that will upload relevant files to S3 after an instance is triggered to shutdown but before it terminates.

How Curalate uses MXNet on AWS for Deep Learning Magic

Productionalizing ECS

In January of last year we decided as a company to move towards containerization and began a migration to move onto AWS ECS. We pushed to move to containers, and off of AMI based VM deployments, in order to speed up our deployments, simplify our build tooling (since it only has to work on containers), get the benefits of being able to run our production code in a sandbox even locally on our dev machines (something you can’t really do easily with AMI’s), and lower our costs by getting more out of the resources we’re already paying for.

Choosing a Deep Learning library for developing and deploying your App/Service

Interest in deep learning is growing and growing and, with it at peak hype right now, a lot of people are looking to find the best deep learning library to build their new app or bring their company into the modern age. There are many deep learning toolkits to choose from ranging from the long used, supported, and robust academic libraries to the new state-of-the-art, industry backed platforms.

R&D At Curalate: A Case Study of Deep Metric Embedding

At Curalate, we make social sell for hundreds of the world’s largest brands and retailers. Our Fanreel product is a good example of this; it empowers brands to collect, curate, and publish social user-generated photos to their e-commerce site. A vital step in this pipeline is connecting the user generated content (UGC) to the product on our client’s web site. Automating this process requires cutting edge computer vision techniques whose implementation details are not always clear, especially for production use cases. In this post, I review how we leveraged Curalate’s R&D principles to build a visual search engine that identifies which of our clients’ products are in user generated photos. The resulting system allows our clients to quickly connect user generated content to their e-comm site, enabling the UGC to generate revenue immediately upon distribution.

Load Testing for Expected Increases in Traffic with Vegeta

At Curalate, our service and API traffic is fairly tightly coupled to e-commerce traffic, so any increase is reasonably predictable. We expect an increase in request rate towards the beginning of November each year, with traffic peaking at 10x our steady rate on Black Friday and Cyber Monday.

Tracing High Volume Services

We like to think that building a service ecosystem is like stacking building blocks. You start with a function in your code. That function is hosted in a class. That class in a service. That service is hosted in a cluster. That cluster in a region. That region in a data center, etc. At each level there’s a myriad of challenges.

From Thrift To Finatra

There are a million and one ways to do (micro-)services, each with a million and one pitfalls. At Curalate, we’ve been on a long journey of splitting out our monolith into composable and simple services. It’s never easy, as there are a lot of advantages to having a monolith. Things like refactoring, code-reuse, deployment, versioning, rollbacks, are all atomic in a monolith. But there are a lot of disadvantages as well. Monoliths encourage poor factoring, bugs in one part of the codebase force rollbacks/changes of the entire application, reasoning about the application in general becomes difficult, build times are slow, transient build errors increase, etc.

Content Based Intelligent Cropping

Square pegs don’t fit in round holes, but what if you have power tools?

Creating an iOS Share Extension for an Ionic App

At Curalate we build a mobile app that allows our clients to use our services on-the-go. It provides an ever-evolving subset of features from our web experience and we ship updates every month or so, but we don’t have any engineers that specialize in mobile development on our team! Ionic enables us to leverage the expertise of our full-stack engineers to ship a product on mobile using client technology that we’re already familiar with - AngularJS. We write well-factored JavaScript in Angular services once for the web, layer on unit tests, then leverage that same code to build an iOS app.

Mechanical Turk Lessons Learned

At Curalate, we constantly dream up big ideas for new products and services. Big ideas that require lots of work. Lots of boring, repetitive, simple work that we honestly do not want to do ourselves. In situations like this we turn to the industry standard for getting other people to do work for you, Amazon Mechanical Turk. Amazon’s “Artificial Artificial Intelligence” service connects requesters to people from across the globe (known as “Turkers”) to complete a set of Human Intelligence Tasks or HITs. We’ve used Mechanical Turk in the past to create labeled datasets for use in our machine learning models to tackle various deep learning problems here at Curalate (such as the Emojini 3000 and Intelligent Product Tagging). In the process we have learned a few lessons that would have saved us a lot of time if known beforehand, so here they are to help if you want to get the most out of your valuable Turk time.

Building a Tracking Pixel in 3 Steps (featuring AWS Kinesis Firehose!)

At Curalate, we need to be able to use data to demonstrate that our products hold value for our clients. One of our products, Fanreel, uses user-generated content to enhance online shopping experiences and product discovery. We record and store usage metrics from Fanreel but we also need to take those usage metrics and connect them to product purchases. If Fanreel analytics were a puzzle, purchase information would be the last piece and historically, Google Analytics or Adobe Omniture served as this last piece. However, every ecommerce site is different so sometimes the intricacies of Google Analytics and Adobe Omniture got in the way.

Programmatic Jenkins jobs using the Job DSL plugin

Jenkins is an incredibly powerful and versatile tool but it can quickly become a maintenance nightmare: jobs are abandoned, lack of standardization, misconfiguration, etc. But it doesn’t have to be this way! By using the Jenkins Job DSL plugin you can take back control of your Jenkins installation.

Zipkin At Curalate

Curalate uses a micro-services infrastructure (SOA) to power its products. As the number of services began to grow, tracking down performance issues became more difficult due to the increasing number of distributed dependencies. To help identify and fix these issues more quickly, we wanted to utilize a distributed tracing infrastructure. The main goals were:

Tips for Starting With Redshift

When beginning to use Amazon Redshift there are always some growing pains. In this blog post we’ll go through 3 tips and tricks we learned from starting up our own Redshift pipeline!

Announcing Curalate's New Muse Page!

The good people at The Muse have made a lovely profile page detailing what it’s like to work at Curalate. We’re extremely proud of the culture we foster, the vision we pursue, and the products we build. You can check out more about the profile at the main Curalate blog.

Hotline Ping: URL availability monitoring built with AWS Lambda and the Serverless Framework

Say you want to track the health of your API. Pingdom is probably your first move. But what if you want to track thousands of endpoints? Suddenly you’re looking at a monthly bill of over $500! So, what if you’re willing to build an API health tracker yourself? This blog post is for you. Not only can we beat $500 per month, we can build our API health tracker for damn near free!

Bridging C++ to Scala with BridJ

At Curalate we’ve moved towards a microservice architecture with each service living in its own git repository. For the most part, we’ve standardized the way we build our Scala projects using Apache Maven to manage dependencies and compilation. This is convenient since any Curalady / Curalad can clone one of our repos and type mvn install at the root with the expectation that everything will compile successfully on the first try. We wanted this same ease of use for our Scala projects that needed access to native libraries and this post explains how we obtained it.

Avoiding Pitfalls with DNS and AWS Elastic Load Balancer

Using ELB for your backend microservices? Seeing intermittent connectivity issues, partial outages across your instances, or other unexplainable failures?

Build and Deploy at Curalate

Since Curalate began three years ago, our build and deploy pipeline has changed immensely. From a manual process run locally on our laptops to an automated system consisting of Jenkins, Packer, Chef, and Asgard, the progression has given us confidence in the system and allowed us to develop and deploy ever faster. In this post I’ll talk about how we build and deploy our code at Curalate. We’ll cover where we started three years ago, where we are now, and what the future holds.

Brewing EmojiNet

Emojis, the tiny pictographs in our phone’s keyboard, have become a important form of communication. Even Oxford Dictionary named 😂 the word of the year for 2015. Though maybe not worth a thousand words, each emoji evokes a much richer response for the reader than boring old text. This makes sense: people communicate visually.

Angular Performance Considerations or: How We Ended Up Detaching $$watchers

When I came to Curalate, our dashboard was a bunch of page-specific jQuery mixed with some helpful libraries. Naturally, this setup became unwieldy as we added more developers to the team and more features to our product. This model became unmaintainable as we planned on more cohesive and complex features and we decided to begin using Angular. The change was welcomed throughout the team and has changed our development process for the better, but migrating to Angular did not come without issues. In this post, I’ll describe the performance bottlenecks we encountered when creating one particularly complex feature using Angular and the methods we employed to solve these issues.

Welcome to the Curalate Engineering Blog!

At Curalate, we’re helping the world’s greatest brands unlock the power of pictures. We’re doing so by building novel, interactive customer experiences and extracting insights and intelligence from billions of consumer engagements every month. If you’re a digital marketer, we’d love to show you a demo of our tools at work.