All articles

A New Way to Think About Data APIs

When I was a kid, we used to play a game called “Mother May I”. In the game, the person who was it, called “Mother”, would stand on one end of the room and the players would stand on the other. The players would shout out things such as, “Mother, may I take one giant step forward?” or “Mother, may I take two scissors steps?” Mother would say either “Yes” or “No.” Players pleaded with Mother in random order. Mother had to say “Yes” more often than “No.” The first player to reach Mother became Mother, and a new game would start. Not exactly the most exciting game in the world, but we were kids so give us a break.

I like to count shit. I’ll count anything I can find. Cracks in the sidewalk. Pennies in a cup.

I also like to make lists. In fact, at one point in my life, combining my two passions, I was maintaining a list of over 350 rules related to how to pick up pennies without incurring bad luck.

Yes. I suffer a bit from OCD (but happily).

Recently, Pedro Gomes made a tweet asking if anyone had a list of the most frequently called function signatures on the Ethereum blockchain.

I wanted to share with you (through a series of charts) what happens when one releases a world-class data scientist such as Ed Mazurek on fresh-baked Ethereum difficulty data.

You get TrueBlarks (that’s a portmanteau of “TrueBlocks” and “R” in case you were wondering). If you don’t know about “R” and R Studio, you should. It’s amazing.

With little to no explanation, I am going to copy and paste the “R” code right next to the chart used to create it. Ask Ed what the code means.

This is a very short take on something I’ve been thinking about for quite a while. “Why do I give a shit about decentralization? Should I compromise?” Here’s why I come down emphatically on the side of “Yes, I give a shit, and no you should never compromise!”.

Blockchain-like technologies (that is, decentralizing, trustless technologies) bring to the world, for the first time in human history, a way to help us solve the prisoner’s dilemma.

When I first heard of Ethereum, I thought to myself “Excellent! I no longer have to keep track of my spending. Everything will be automated.”

Can you say “misconception”?

Not only is keeping track of Ethereum spending way more difficult than it should be, it’s actually so difficult that I stopped spending ether all together about two years ago. (The price of ether had a lot to do with it as well.)

I had a wonderful experience at EdCon in Toronto this week. Mostly because of all the great new people I met and the many people I reacquainted myself with. The Ethereum community is freakin’ cool.

Here’s an example. In between speakers, the person sitting in front of me stood up, turned around, looked down at his seat and, using his pointing finger, counted the number of seats between him and the left isle. He then counted the number of seats between himself and the right isle. Finding that the right-most path was shorter, he existed the auditorium in that direction. He didn’t care which way he went — he cared only about being efficient.

I was recently included in a discussion about why it’s so difficult to get ERC20 token balances (and other data) from the Ethereum blockchain. I thought I’d take a crack at answering the question.

I think it comes down to four words: “Incoming”, “Internal”, “Transactions”…and “Decentralization.” I’ll focus on each word individually.

Transaction

I’ll start with the easiest word first: “Transaction.”

Go into a bagel store. Buy a dozen bagels and get a receipt. Everything you need to know about your transaction is on the receipt. Date, amount, sender (from) and recipient (to). This is the same information you’ll find on every Ethereum transaction.

I recently wrote a piece discussing how I defeated the Ethereum DDos attack using TrueBlocks. Doing this was important because it freed me from the pain of a slow RPC. Speed allows me to analyze the Ethereum data iteratively. I can find more interesting stuff.

The Ethereum dataset is big (and growing). I want to be able to scan through the entire thing. I want to be able to do this on a laptop. This last fact makes it impossible for me to create a separate, independent copy of the data. If I want to do data analysis, I have to do it against the node’s data directly.

I spend a lot of time looking at historical Ethereum transactional data. I do this by scanning the chain using TrueBlocks.

If you’ve ever done this, you will be familiar with a certain set of transactions that take a very long time to process. These transactions happened between blocks 2,286,910 and 2,717,576. They are a pain in my a$$. See here.

In a surprisingly effective attack, some evil genius took advantage of an underpriced opcode to create millions of dead Ethereum accounts. This had the effect of significantly bloating the state database, but more importantly for our purposes it created tons of transaction traces.

I am so deep down the rabbit hole of Ethereum data that I sometimes forget there is such a thing as the real world. And I care exactly zero if I never come up. Don’t ask me why. I’m pretty sure there’s no good reason.

Maybe my obsession with this has to do with something that happened to me in high-school. My guidance counselor administered a test intended to identify which career I should pursue. She watched me take the test and then carefully scored my answers. When she was done, she looked across the table and said, “You should look for careers that have nothing to do with people.” The remaining options for me where making things (factory work) and information processing. I took the latter path.