Importance of changelogs

It’s very important to keep a changelog and be disciplined about maintaining it as your product progresses. This applies to both product management and software development/management processes.

I want to mention a great source that talks, describe and almost accepted as a standard in open source community for changelogs:

https://keepachangelog.com

What is a changelog?

A changelog is a file that contains a curated, chronologically ordered list of notable changes for each version of a project.

Why keep a changelog?

To make it easier for users and contributors to see precisely what notable changes have been made between each release (or version) of the project.

Who needs a changelog?

People do. Whether consumers or developers, the end-users of software are human beings who care about what’s in the software. When the software changes, people want to know why and how.

An example Changelog (from Stretchly)

Github renders this markdown changelog beautifully: https://github.com/hovancik/stretchly/blob/master/CHANGELOG.md

Using Cloudinary for image cloud storage with image transformations in your NodeJS express app in Heroku

Here we are with another article about the development aspect of photo/image management (storage, serving, retrieval). I’ve previously (right before this article) wrote about Authenticating and Getting Instagram Photos in NodeJS/Express application. This story is about manually storing, handling upload, download and serving static photo/image using a CDN service called Cloudinary.

Content should be separate than the application

We’re (web/back-end/front-end developers) building apps, sites in many different ways (different platforms, languages, stack). One thing very common and old school is that everything on the site is organized in the same bucket. So when we code and deploy a site, it’s HTML, CSS, back-end code, images, videos, fonts, etc… all in the same place. Now we have distributed deployment systems with having multiple instances of our application on different web servers too. Which made us do a soft transition to keep common files (that can be changed) like uploads folders in block storages like s3 or azure blob… But it still doesn’t do the full justice that both static and dynamic content of an application/website should be completely separated than the application code. This is not a new practice but it’s a practice that can be missed so easily.

It’s so easy to leave an image that is used in a blog post within the codebase (which is wrong). A static content that will not be used to render your page on the back-end ideally doesn’t belong to the place where you store your application (code). Yet, it shouldn’t be served (or requested) from the same servers which are responsible (only should be responsible) for rendering and serving your pages. Tiring your web server with serving images or compiled CSS is not optimal. This will also affect your site’s performance that everything is coming from the same source. Having distributed sources will make your browser to manage parallel downloads of your site’s resources faster. No brainer!

We’ll not get into the techniques of how to separate these different things in different places in this article, but we’ll talk about images specifically.

Images need a big brother

This was a novelty in the past where we wanted to have multiple sizes of an image/photo so we can economically request the right size in different pages – example: get 100px width thumbnail in the page where we show photos in a grid, show the 500px width version on the lightbox, and link out to the original photo in the “download” button. This makes total sense right? Strategizing how to have the different versions ready on your server (or CDN) is another thing. So many self-solutions you can use or code it up. But from user experience (admin/editor) standpoint, nobody wants to do this manually or even automatically but wait for the server to resize and prepare these versions when uploading a single photo to your CMS/app/back-end. To me, that is a one-way road. I only want to upload an image and only wait the time takes the file transfer from my device to the server. Done!

What is Cloudinary and should I use it?

Cloudinary is that big brother and storage and server together. Smart CDN for images and videos. It has a pretty decent free package that will be enough for almost all personal, experimental and small projects. If you have decent size traffic, you may think to pay or optimize your solution with Cloudinary.

Cloudinary hosts and serves images for you. It’ll be your storage bucket that also has many out of box solutions for known CMSs like WordPress. I like the API/SDK route which they have SDKs and well-designed API for almost all platforms. We’ll play with NodeJS below.

The magic cloudinary has that is compelling that it can do so many varieties of transformations on your images on the fly (and caches them). Basic things like color filters, crop, resize, rotate etc… But the real thing is where they have face recognization that you can create square avatars with intelligently telling cloudinary to give you the face focused position in the center on your circle avatar with transparent png background and have 2px border around circle cropped avatar. All of it happens over URL parameters. True magic. I haven’t digg the video side of this but I read bunch of smart stuff on the streaming site which is also worth considering cloudinary as one-stop-shop for visual static assets.

Add Cloudinary service on your heroku application

Adding a service to heroku application is very easy and mostly done in command line. In order to create a new cloduinary service as add-on to your application, in your application folder run:

This command will create a new cloudinary account linked to your heroku account and add cloduinary credentials to your heroku config – environment variables. You can see and copy the variables to your local .env file with

Using it on nodejs/express app

Install first:

server.js

See this example on github: https://github.com/mfyz/heroku-cloudinary-uploads-example

WordPress and other sites

Cloudinary has SDKs and official plugins to well-accepted platforms like WordPress. Check out their official documentation about the ent libraries and plugins


You can also use my invitation link to give me extra free credits: https://cloudinary.com/invites/lpov9zyyucivvxsnalc5/cdlhm6z9q63gdufko1kj

Single JavaScript file node/express/Instagram authentication (OAuth) and get user photos

Get Ready

Register your Instagram app in their developer portal and obtain client id and client secret keys. To do that, follow the steps below:

  • Go to IG Developer page: https://www.instagram.com/developer/
  • Click “Register your application” button (if you are not logged in, you will be asked to log in on this step)
  • Fill all fields. The only field you need to pay attention to is the “valid redirect URLs”. This is where your app is publicly hosted. Below, we will create a URL on the application to capture Instagram authentication after the user goes to the Instagram page for permission dialog, then comes back to this URL. It’ll be something like https://yoursite.com/instagram/callback
  • Once you register the app, the page will display client id and secret. Keep this ready on the next steps.

Code it up

Let’s set up the plain node.js and express the application.

First, install the required packages:

index.js (or server.js)

Deploy and run

Either locally or after you place it on your server, run:

Tip 1: use “forever” on your server to run this application permanently.

Tip 2: For experimental purposes, you can run this app on your local and have a tunneling tool like “ngrok” to open your local port to the public with a quick domain assignment. Ngrok will provide a URL (random subdomain on their domain), you have to update the IG developer app’s settings to add this domain as a valid redirect URL, otherwise, after this app redirects user for authentication to Instagram, it will give an error.

Get the real thing

The code above in this article was a quick and dirty version. I put the little bit more proper express application version on Github. It uses pug for its views and has proper layout/content block separation as well.

https://github.com/mfyz/express-passport-instagram-example

WDS (World Domination Summit)

WDS - Welcome Adventurer

An unusual gathering happens every year in Portland, OR. Well at least it is going to happen one last time next year.

WDS is an event bringing radical, open minds to share world dominating ideas that creates impact on multiple fronts. General theme is to be, think unconventional, to explore different view points on both common and uncommon topics like housing, work, living free, being creative. All for having a purpose in life and work, being impactful to human kind, supporting communities. WDS built it’s own community too.

Snapshots from the WDS

Having so many interesting walks of lives gathered and opening themselves up to each other, understand each other, support each other, be vulnerable, share big ideas, seek ways to be useful, be better human beings. Many radical meetups, discussions takes place throughout the conference, and out of it, a lot of sparks happens. People open, listen, think and change in this conference.

It was exactly like what expected, to me. I didn’t radically change, but I made great friends, heard a lot of inspiring stories, took tons of notes, questioned things a lot…

I’ve observed (didn’t attend for dumb excuses) for 2 years. Then this year (2019) attended to the conference. I will attend the next (and the last one) as well.

https://worlddominationsummit.com

How I use Slack as my Dashboard

Slack has become how many teams communicate internally. In essence, select is an old-school instant messaging platform that has existed from IRC days. What makes slack so powerful is its integrations with other services. Almost all popular services integrate with Slack with a click of a button.

I wanted to talk about a few different angles I utilize slack in my personal and team accounts.

Talk to humans

Slack is the centerpiece on our remote/distributed team on multiple timezones and multiple cities/countries/continents because communication is the centerpiece on remote teams and slack is doing a great job to give a plain tool to communicate. Of course, it’s not the only tool we use to communicate but it’s the most frequent one.

Speaking of communication, communication is not exclusive to humans in our scenario. We also communicate with bots, servers, services, tools, etc… Fortunately, big names (Google, Trello etc) is already nicely integrated with Slack. So we use their apps/bots to talk or listen to them on slack. In some cases, we use slack as our source to talk to these services.

Listen to no-human activity without getting distracted from your slack routine

Slack can be a great “monitoring” platform for keeping eye on things (everything) from a single point of view. This makes slack different than a just chatting app. You can set up pretty much any “notification sending” tool/app to send these notifications on slack. Things like your website’s uptime status, order tracking, new tweets/IG photos/daily news, new blog posts from your favorite blogger… Anything that can be received as email can be set up be redirected to a slack notification in your own categorization skills.

Even “Track Who Goes To Space With IFTTT“ which I talked about it before. The way I was tracking on a slack channel too.

There is also a great article written in Smashing Magazine about using using slack to monitor your app that exhibits this use case well.

Make non-humans to listen you through slack – run your stuff / take actions

We do this all the time without slack. Thing like opening your calendar app and creating new event/reminder/meeting and invite others. Or open amazon app and buy stuff. Go to Trello and create a task to yourself or your team member. Or share a dropbox file. We do these things on our devices with manual steps, using each service’s apps/tools or tools that are designed for that purposes.

But slack brings standardization to these things. A single interface to make these happen with making bots listen to you and do these stuff for you. Some of these “actions” are given in plain English (or your language) or in most cases, though slack’s rich message features like buttons or slash commands. Few things we did and doing regularly on our team: like creating meetings with meekan (scheduling bot) or create Trello cards from slack while we discuss something with the team without getting distracted to open Trello and create cards.

There are many other cases we use slack to “take actions” within slack. The beauty of this is you can make slack very smart with bots that trigger things to services you use through bots. Also extendable that we can write bots to do things that are not provided with existing bots and services or custom stuff. Or even write bots for new things we create.

Why I don’t like wearables? (that are attached to your body)

Men wearing wearable wristband (apple watch)

Prepare yourself for a cynical opinion 🙂

I find wearables (mostly wristbands and watches) too annoying and mostly useless. There is a stereotype I’ll follow in this article which, when someone says wearables, I think (and mostly criticize) wristbands. Let me put it out there that there are many versions of wearables that you can wear and carry. I’m happy and ok with most wearables that don’t need to be physically attached to the body and don’t need to be charged every day or every other day. In general, good tech is the ones that you don’t feel any different or do any different than how you live right now. Let’s get the stereotype wristband and likes on the table.

Activities? Meh!

I know I lot of people use to track their activities, like walking, running but I’m sure the majority of the people are not professionals or taking the activity tracking seriously. In my case, everybody I know wearing apple watches are not using them for that. Kind a novelty to have your walking tracked. You either walk or don’t. It’s like a wearable will make me go to swimming more often or not, of course, it will not.

Notifications – God no!

Wearables are like demons in my head. Well, my phone is actually already like that. Wearables are mostly configured to notify you by default when you open them in the box. I know they are changeable, which I tried to make a silent apple watch. But then why am I wearing it, what’s the purpose of it now? I found only 2 passive notifications helped me when I was trying out different wearables:

  • Inactivity alert, where if you sit and don’t move for an hour, you get a nudge that will remind you to move your a** from the chair – which is great and impactful for sure. Instead, I use screen timeout tools that do the same thing (mostly).
  • Good old timer (wrist version of the kitchen timer). This is used for anything but in most cases my pomodoros (or focus blocks, way to GTD). I set the timer with Siri (in apple watch case) and get to work until I get the nudge. This was the only “real” use case I had – but for having a timer at $350+ cost is just dumb.

Health – Yes!

For seniors, wearables that do consistent hearth monitoring throughout the day is probably the most impactful way to use wearables in my opinion.

Sleep? Hell yes but no 😞!

What I loved on body-attached-wearables from the beginning was sleep tracking. But unfortunately “in theory”. Nobody got this right. Jawbone Up was my first and beloved sleep tracker worked “the best” but had a lot of room to improve. Then Jawbone stopped improving this feature (maybe nothing left to improve). Even though the hardware was fragile and gets broken after few months, I was happy to keep buying same hardware many times (I swear I had 10+ of same and different versions of Jawbone Up – I actually still have unopened box ones). But then Jawbone discontinues to sell them (well, I have 3 unopened ones, so I thought I was good for couple years), then they shut down the servers which made the mobile app to not work at all (because it’s a cloud/API based app) which basically made all Jawbone wearables garbage.

Then every single wearable copied but copied the shittiest version of sleep tracking including Apple. Even if Apple was nailed it, it’s just moronic that you have to charge your apple watch pretty much every day – and any real opportunity you have is when you sleep which you are (in practice) forced to not wear it. Please, someone, get this right…

If you really have to…

put something on your wrist to be cool or maybe really tracking your activities, please don’t make it rain! It’s just a waste. Now there is a sea of wearables does the exactly same stuff every other one does as low as $20 on Amazon. I recently tried Xiaomi’s 4th gen wearable which is pretty good – does my timer and alarm functions well (that’s enough for me but if you are interested, it does all the other things too), and I only need to charge almost once a month (well, I only wear it when I sleep).

Analytics Data on SQL Database – Best database and table design for billions of rows of data

This is not an article that I am writing but I’m mostly quoting a great gem on a stack overflow answer I came across when I was researching a DIY way to store and create analytics reports for a small to medium size project. The project’s type doesn’t matter because this is a generic problem and great solution.

Why not use analytics tools/services?

I am in constant search of the better alternatives or simpler versions of the solutions we use at my team. We certainly use many services and tools from open source to licensed software. But I still choose to understand, know and be able to apply these solutions by myself on a custom solution where I have full control over the data, output and user experience.

So I casually read and research how others approach the issues or queries wander in my mind.

Then I stumbled upon this stack overflow thread with a brilliant answer that contains steps to try out from scratch that I suggest any engineer to just try and play on their own time.

PostgreSQL and BRIN indexes

To create a sample table with 1.7 billion rows of a sample sensor data (temperature read from the sensor with timestamps in the logs):

So it took 22min to create the table. Largely, because the table is a modest 97GB. Next, we create the indexes,

It took a good long while to create the indexes too. Though because they’re BRIN they’re only 2-3 MB and they store easily in ram. Reading 96 GB isn’t instantaneous, but it’s not a real problem for my laptop at your workload.

Now we query it.

Update with timestamps

Here we generate a table with different timestamps in order to satisfy the request to index and search on a timestamp column, creation takes a bit longer because  to_timestamp(int) is substantially more slow than  now() (which is cached for the transaction)

Now we can run a query on a timestamp value instead,,

Result:

So in 83.321 ms we can aggregate 86,401 records in a table with 1.7 Billion rows. That should be reasonable.

Hour ending

Calculating the hour ending is pretty easy too, truncate the timestamps down and then simply add an hour.

It’s important to note, that it’s not using an index on the aggregation, though it could. If that’s your typical query you probably want a BRIN on  date_trunc('hour', tsin) therein lies a small problem in that  date_trunc is not immutable so you’d have to first wrap it to make it so.

Partitioning

Another important point of information on PostgreSQL is that PG 10 bring partitioning DDL. So you can, for instance, easily create partitions for every year. Breaking down your modest database into minor ones that are tiny. In doing so, you should be able to use and maintain btree indexes rather than BRIN which would be even faster.

This is a great answer to the topic around working with analytics data on SQL databases. Finally thinking about table partitioning is always a good plan-ahead strategy for any data gets over millions and have distributed data around timestamps.

Reference: Best database and table design for billions of rows of data

Make your car smarter with Automatic

After I got my last car about 8 months ago, I’ve invested some money to make my car smarter like getting Apple car play installed. One of the things I did was to install a mini device to my car that makes my car a little bit smarter. The device name is Automatic.

This tiny gadget connects to the car’s diagnostic port which it receives it’s power and read and monitor car’s data like engine lights and stuff. The device contains GPS tracker and a sim card to continuously have low-speed connectivity like GPRS. This is more than enough to store it’s monitoring data in the cloud.

Here is the cool stuff you can do with having automatic on your car:

  • Parking tracking – probable the best feature. I hate the idea of trying to remember where I parked in the street. This removes the need for that. Open the app and check where the car is 🙂
  • Track when you started and finish driving. It’s like anti-theft alarm. You get a notification if someone starts the engine – sweet
  • Track the whole tour you drove. It draws the driving path on the map and you can see your driving history like uber receipts.
  • These driving records also contain avg speed, gas consumption…
  • Driving style – how often you stop. how aggressive you accelerate…
  • IFTTT integration – this opens the possibilities 100 times. You can set triggers like driving away from am geofence, or arriving at the geofence. This allows setting smart behaviors like Turn on garage lights when I arrive home.

It’s a little bit of a novelty features but if you are data nerd like me, it’s automatic tracking of your driving data which is enough value to me.

https://automatic.com

Using Heroku for a quick development environment

Heroku is an industry-changing service that is established in 2007. It transformed how developers create and deploy apps today. With its add-ons marketplace, Heroku became the development hub that you can easily enable 3rd party cloud services. These services can be in many different categories that a web application may require. From database services, caching, image processing to mail delivery and so on…

Heroku supports many modern development languages that are actively used with big communities like PHP, nodejs, ruby, python, go, java… The beauty of the Heroku applications is that, managed by Heroku and very very easy to understand. They are also very easy to scale, deploy apps in Heroku infrastructure… All Heroku apps are deployed to given app name’s subdomain under herokuapp.com or can be easily set to have a custom domain for free.

Essentially, Heroku runs on a command line interface and an internal git repository to manage versions of your code. When you set up a new project folder, Heroku CLI tool registers your app and assigns a git repository. Heroku CLI doesn’t initiate git repository on your folder, so if it’s a non-git folder, you need to git init on your project folder first.

I highly suggest all developers adapt Heroku in their workflow, at least for the sandbox & playground purposes.

I have created some boilerplate repositories in the past:

Time to take a break from your computer

I firmly believe that having regular breaks from long work sessions is a necessity. We work sitting long hours without much movement. It’s often we forget to take long or short breaks when we work on a task that occupies our brain. Also sometimes it’s hard to break a focus session.

Without a tool reminding us of the breaks, it’s very difficult to track when we started working and when to take a break. GTD methods like Pomodoro helps to track these moments but even with planned natural work blocks we miss taking a break from the screen.

I started to use a tool to remind break time with a schedule. First I used a paid app called Timeout.app. Then one of my colleagues found the open source alternative of the same tool. https://github.com/hovancik/stretchly

I’ve forked and modified the look and feel and some of the element’s positions for my own taste https://github.com/mfyz/stretchly

I highly suggest a timeout app like this one to be embedded in the workflow. Now I take much more regular breaks with the help of a tool like this. Sometimes I feel it becomes annoying because I feel I’m very focused. I don’t want to break the focus session but I also acknowledge the importance and the value of giving opportunity and space to our minds with these breaks.

The app is pretty straight forward to use. You set micro and regular breaks with an interval. I take 10 seconds micro break in every 10 minutes. Every after 2 micro breaks, I take a regular break of 5 minutes (every 30 minutes)

https://github.com/mfyz/stretchly