标签：编程资源

2018年4月1日：开源日报第24期

1 4 月, 2018

每天推荐一个 GitHub 优质开源项目和一篇精选英文科技或编程文章原文，欢迎关注开源日报。交流QQ群：202790710；电报群 https://t.me/OpeningSourceOrg

今日推荐开源项目：《汇总资源宝库awesome》

推荐理由：本项目提供了大量精选的阅读材料，旨在帮助后台开发者们能够借助项目中各个文献的思路做出一个具有延展性，可用性，稳定性的后台。这个概念虽然模糊，但是借由著名工程师（Martin Fowler，Robert C. Martin，Tom White等）和高质量资源（highscalability.com，infoq.com等）的诠释，相信大家能从中理解并学到有用的东西。

项目将各类文献整理，涵盖各个方面。主要分为原则、可扩展性、稳定性、其他方面、会谈、图书七个板块。包括了异常处理，数据库策略等内容。

什么是 awesome 项目

这是 github 上一个有名的系列，项目名一般为 awesome-xxx ，这个系列一般是用来收集学习、工具、书籍类相关的项目，总之，这个系列一定能大大的满足您的收集癖.

例如：

awesome-python：https://github.com/vinta/awesome-python

Python相关资源汇总

Awesome-Linux-Software: https://github.com/LewisVo/Awesome-Linux-Software

Linux上的好的软件及相关资源的汇总

当然，还有一份大汇总列表，它的项目名就叫做awesome，链接：

https://github.com/sindresorhus/awesome

所以说，如果您不知道有什么资源可以用时，大可以在awesome 中查找，相信还是有很大的概率能得到您想要的东西的

创建一个awesome 项目（列表）

如果您有了与大家一起分享有趣或有价值的资源的想法，那不妨建立一个awesome 列表让更多的人一起参与进来，当然，请您务必参考awesome 提出的标准，链接：

https://github.com/sindresorhus/awesome/blob/master/create-list.md

作者：Benny Nguyen

新加坡软件工程师（分布式系统&大数据）

新加坡人

在越南国立大学获得本科学位，在新加坡过哦立大学获得计算机硕士学位

技能专长如算法，java，c++，c#等多达25项

今日推荐英文原文：《Ten Machine Learning Algorithms You Should Know to Become a Data Scientist》作者：Shashank Gupta

原文链接：https://towardsdatascience.com/ten-machine-learning-algorithms-you-should-know-to-become-a-data-scientist-8dc93d8ca52e

推荐理由：机器学习热度持续不减，一方面有市场炒作，一方面其本身的魅力令让难当，尤其是加持数据之美，海量的数据加上机器学习算法真的可以碰撞出很赞的火花，这篇文章就给大家介绍了10个机器学习算法，如果你想成为一个数据科学家？这篇文章正好。

Ten Machine Learning Algorithms You Should Know to Become a Data Scientist

Machine Learning Practitioners have different personalities. While some of them are “I am an expert in X and X can train on any type of data”, where X = some algorithm, some others are “Right tool for the right job people”. A lot of them also subscribe to “Jack of all trades. Master of one” strategy, where they have one area of deep expertise and know slightly about different fields of Machine Learning. That said, no one can deny the fact that as practicing Data Scientists, we will have to know basics of some common machine learning algorithms, which would help us engage with a new-domain problem we come across. This is a whirlwind tour of common machine learning algorithms and quick resources about them which can help you get started on them.

1. Principal Component Analysis(PCA)/SVD

PCA is an unsupervised method to understand global properties of a dataset consisting of vectors. Covariance Matrix of data points is analyzed here to understand what dimensions(mostly)/ data points (sometimes) are more important (ie have high variance amongst themselves, but low covariance with others). One way to think of top PCs of a matrix is to think of its eigenvectors with highest eigenvalues. SVD is essentially a way to calculate ordered components too, but you don’t need to get the covariance matrix of points to get it.

This Algorithm helps one fight curse of dimensionality by getting datapoints with reduced dimensions.

Libraries:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.svd.html

http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html

Introductory Tutorial:

https://arxiv.org/pdf/1404.1100.pdf

2a. Least Squares and Polynomial Fitting

Remember your Numerical Analysis code in college, where you used to fit lines and curves to points to get an equation. You can use them to fit curves in Machine Learning for very small datasets with low dimensions. (For large data or datasets with many dimensions, you might just end up terribly overfitting, so don’t bother). OLS has a closed form solution, so you don’t need to use complex optimization techniques.

As is obvious, use this algorithm to fit simple curves / regression

Libraries:

https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.lstsq.html https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.polyfit.html

Introductory Tutorial:

https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/linear_regression.pdf

2b. Constrained Linear Regression

Least Squares can get confused with outliers, spurious fields and noise in data. We thus need constraints to decrease the variance of the line we fit on a dataset. The right method to do it is to fit a linear regression model which will ensure that the weights do not misbehave. Models can have L1 norm (LASSO) or L2 (Ridge Regression) or both (elastic regression). Mean Squared Loss is optimized.

Use these algorithms to fit regression lines with constraints, avoiding overfitting and masking noise dimensions from model.

Libraries:

http://scikit-learn.org/stable/modules/linear_model.html

Introductory Tutorial(s):

https://www.youtube.com/watch?v=5asL5Eq2x0A

https://www.youtube.com/watch?v=jbwSCwoT51M

3. K means Clustering

Everyone’s favorite unsupervised clustering algorithm. Given a set of data points in form of vectors, we can make clusters of points based on distances between them. It’s an Expectation Maximization algorithm that iteratively moves the centers of clusters and then clubs points with each cluster centers. The input the algorithm has taken is the number of clusters which are to be generated and the number of iterations in which it will try to converge clusters.

As is obvious from the name, you can use this algorithm to create K clusters in dataset

Library:

http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html

Introductory Tutorial(s):

https://www.youtube.com/watch?v=hDmNF9JG3lo

https://www.datascience.com/blog/k-means-clustering

4. Logistic Regression

Logistic Regression is constrained Linear Regression with a nonlinearity (sigmoid function is used mostly or you can use tanh too) application after weights are applied, hence restricting the outputs close to +/- classes (which is 1 and 0 in case of sigmoid). Cross-Entropy Loss functions are optimized using Gradient Descent. A note to beginners: Logistic Regression is used for classification, not regression. You can also think of Logistic regression as a one layered Neural Network. Logistic Regression is trained using optimization methods like Gradient Descent or L-BFGS. NLP people will often use it with the name of Maximum Entropy Classifier.

This is what a Sigmoid looks like:

Use LR to train simple, but very robust classifiers.

Library:

http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

Introductory Tutorial(s):

https://www.youtube.com/watch?v=-la3q9d7AKQ

5. SVM (Support Vector Machines)

SVMs are linear models like Linear/ Logistic Regression, the difference is that they have different margin-based loss function (The derivation of Support Vectors is one of the most beautiful mathematical results I have seen along with eigenvalue calculation). You can optimize the loss function using optimization methods like L-BFGS or even SGD.

Another innovation in SVMs is the usage of kernels on data to feature engineer. If you have good domain insight, you can replace the good-old RBF kernel with smarter ones and profit.

One unique thing that SVMs can do is learn one class classifiers.

SVMs can used to Train a classifier (even regressors)

Library:

http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html

Introductory Tutorial(s):

https://www.youtube.com/watch?v=eHsErlPJWUU

Note: SGD based training of both Logistic Regression and SVMs are found in SKLearn’s http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html , which I often use as it lets me check both LR and SVM with a common interface. You can also train it on >RAM sized datasets using mini batches.

6. Feedforward Neural Networks

These are basically multilayered Logistic Regression classifiers. Many layers of weights separated by non-linearities (sigmoid, tanh, relu + softmax and the cool new selu). Another popular name for them is Multi-Layered Perceptrons. FFNNs can be used for classification and unsupervised feature learning as autoencoders.

Multi-Layered perceptron

FFNN as an autoencoder

FFNNs can be used to train a classifier or extract features as autoencoders

Libraries:

http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier

http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html

https://github.com/keras-team/keras/blob/master/examples/reuters_mlp_relu_vs_selu.py

Introductory Tutorial(s):

http://www.deeplearningbook.org/contents/mlp.html

http://www.deeplearningbook.org/contents/autoencoders.html

http://www.deeplearningbook.org/contents/representation.html

7. Convolutional Neural Networks (Convnets)

Almost any state of the art Vision based Machine Learning result in the world today has been achieved using Convolutional Neural Networks. They can be used for Image classification, Object Detection or even segmentation of images. Invented by Yann Lecun in late 80s-early 90s, Convnets feature convolutional layers which act as hierarchical feature extractors. You can use them in text too (and even graphs).

Use convnets for state of the art image and text classification, object detection, image segmentation.

Libraries:

https://developer.nvidia.com/digits

https://github.com/kuangliu/torchcv

https://github.com/chainer/chainercv

https://keras.io/applications/

Introductory Tutorial(s):

http://cs231n.github.io/

https://adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/

8. Recurrent Neural Networks (RNNs):

RNNs model sequences by applying the same set of weights recursively on the aggregator state at a time t and input at a time t (Given a sequence has inputs at times 0..t..T, and have a hidden state at each time t which is output from t-1 step of RNN). Pure RNNs are rarely used now but its counterparts like LSTMs and GRUs are state of the art in most sequence modeling tasks.

RNN (If here is a densely connected unit and a nonlinearity, nowadays f is generally LSTMs or GRUs ). LSTM unit which is used instead of a plain dense layer in a pure RNN.

Use RNNs for any sequence modelling task specially text classification, machine translation, language modelling

Library:

https://github.com/tensorflow/models (Many cool NLP research papers from Google are here)

https://github.com/wabyking/TextClassificationBenchmark

http://opennmt.net/

Introductory Tutorial(s):

http://cs224d.stanford.edu/

http://www.wildml.com/category/neural-networks/recurrent-neural-networks/

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

9. Conditional Random Fields (CRFs)

CRFs are probably the most frequently used models from the family of Probabilitic Graphical Models (PGMs). They are used for sequence modeling like RNNs and can be used in combination with RNNs too. Before Neural Machine Translation systems came in CRFs were the state of the art and in many sequence tagging tasks with small datasets, they will still learn better than RNNs which require a larger amount of data to generalize. They can also be used in other structured prediction tasks like Image Segmentation etc. CRF models each element of the sequence (say a sentence) such that neighbors affect a label of a component in a sequence instead of all labels being independent of each other.

Use CRFs to tag sequences (in Text, Image, Time Series, DNA etc.)

Library:

https://sklearn-crfsuite.readthedocs.io/en/latest/

Introductory Tutorial(s):

http://blog.echen.me/2012/01/03/introduction-to-conditional-random-fields/

7 part lecture series by Hugo Larochelle on Youtube: https://www.youtube.com/watch?v=GF3iSJkgPbA

10. Decision Trees

Let’s say I am given an Excel sheet with data about various fruits and I have to tell which look like Apples. What I will do is ask a question “Which fruits are red and round ?” and divide all fruits which answer yes and no to the question. Now, All Red and Round fruits might not be apples and all apples won’t be red and round. So I will ask a question “Which fruits have red or yellow color hints on them? ” on red and round fruits and will ask “Which fruits are green and round ?” on not red and round fruits. Based on these questions I can tell with considerable accuracy which are apples. This cascade of questions is what a decision tree is. However, this is a decision tree based on my intuition. Intuition cannot work on high dimensional and complex data. We have to come up with the cascade of questions automatically by looking at tagged data. That is what Machine Learning based decision trees do. Earlier versions like CART trees were once used for simple data, but with bigger and larger dataset, the bias-variance tradeoff needs to solved with better algorithms. The two common decision trees algorithms used nowadays are Random Forests (which build different classifiers on a random subset of attributes and combine them for output) and Boosting Trees (which train a cascade of trees one on top of others, correcting the mistakes of ones below them).

Decision Trees can be used to classify datapoints (and even regression)

Libraries

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html

http://xgboost.readthedocs.io/en/latest/

https://catboost.yandex/

Introductory Tutorial:

http://xgboost.readthedocs.io/en/latest/model.html

https://arxiv.org/abs/1511.05741

https://arxiv.org/abs/1407.7502

http://education.parrotprediction.teachable.com/p/practical-xgboost-in-python

TD Algorithms (Good To Have)

If you are still wondering how can any of the above methods solve tasks like defeating Go world champion like DeepMind did, they cannot. All the 10 type of algorithms we talked about before this was Pattern Recognition, not strategy learners. To learn strategy to solve a multi-step problem like winning a game of chess or playing Atari console, we need to let an agent-free in the world and learn from the rewards/penalties it faces. This type of Machine Learning is called Reinforcement Learning. A lot (not all) of recent successes in the field is a result of combining perception abilities of a convent or a LSTM to a set of algorithms called Temporal Difference Learning. These include Q-Learning, SARSA and some other variants. These algorithms are a smart play on Bellman’s equations to get a loss function that can be trained with rewards an agent gets from the environment.

These algorithms are used to automatically play games mostly :D, also other applications in language generation and object detection.

Libraries:

https://github.com/keras-rl/keras-rl

https://github.com/tensorflow/minigo

Introductory Tutorial(s):

Grab the free Sutton and Barto book: https://web2.qatar.cmu.edu/~gdicaro/15381/additional/SuttonBarto-RL-5Nov17.pdf

Watch David Silver course: https://www.youtube.com/watch?v=2pWv7GOvuf0

These are the 10 machine learning algorithms which you can learn to become a data scientist.

You can also read about machine learning libraries here.

We hope you liked the article. Please Sign Up for a free ParallelDots account to start your AI journey. You can also check demo’s of our APIs here.

每天推荐一个 GitHub 优质开源项目和一篇精选英文科技或编程文章原文，欢迎关注开源日报。交流QQ群：202790710；电报群 https://t.me/OpeningSourceOrg
2018年3月27日：开源日报第19期

27 3 月, 2018
每天推荐一个 GitHub 优质开源项目和一篇精选英文科技或编程文章原文，欢迎关注开源日报。交流QQ群：202790710；电报群 https://t.me/OpeningSourceOrg

今日推荐开源项目：《Flow——针对JavaScript的静态错误检查器》

推荐理由：Flow是一款针对JavaScript的静态错误检查器，可以在Mac OS X、Linux (64-bit)或是Windows (64-bit)中运行。

Flow的优势
- 安装便捷（在Mac OS X系统，你可以通过Homebrew或OCaml OPAM 包装安装Flow；而在Windows的安装会略微繁琐一点）
- 使用简单
- Flow可以把它的剖析器编译进JavaScript里
详见官方文档：https://flow.org/en/docs/

关于glow

glow 的作用就不赘述了，今天主要说说glow 的安装与使用：
1. 首先，你需要安装 flow（ glow 是基于 flow 的）
2. 然后，你有了两个选择：
- 全局安装glow ： yarn global add glow
安装成功：
- 只在你要的项目中安装glow ：在项目路径下，输入 yarn add –dev glow
安装成功会出现类似画面：
- 然后你便可以愉快的使用glow 了，而使用也只是在flow 的基础上改为执行 glow
测试代码：

运行 flow ：

运行 glow ：

关于作者

詹姆斯·凯尔：http://thejameskyle.com/

今日推荐英文原文：《How to Hack an Engineer》作者：Nemild

原文链接：https://www.nemil.com/musings/hack-an-engineer.html

推荐理由：这篇文章事实上也是一个 GitHub repo，它收集了很多宝贵的资源，如果你想做一个好工程师或者黑客的话，这些宝贵的资源应该可以帮到你。

How to Hack an Engineer

This essay introduces a Github repo to catalog all the ways that engineering inputs can be influenced. Contributions are welcome.
In 2015, I found myself talking to a smart entrepreneur who had just raised millions of dollars in venture capital from a prominent venture capital (VC) firm. He was flying high after getting substantial press and recruiting a number of early employees. His business also wholly depended on a popular open source project.

As part of his strategy, the CEO gave a core open source developer in the project equity. With this, he boasted that he would get critical information sooner, influence the roadmap for his company’s benefit, and recruit developers from the project to his company.¹

I was doubtful this connection would ever be publicized by the open source developer. Would he speak glowingly about the company at a conference, despite private reservations? How would he influence the decisions of his fellow developers? Would developers early in their career follow his missives without adequate reflection?²

Computer engineers regularly deal with exploits on their technical systems, where an external party takes advantage of a bug or vulnerability. An “exploit” or social engineering of our personal information sources — motivated parties influencing subreddits, conferences, meetups, blogs, journalists, professors, industry analysts — should be perceived similarly. As this example shows, even your peers can be influenced.

As engineers, it’s valuable to develop the antibodies that enable us to make the best decisions — both for our teams and for our own careers — despite the desires of these motivated parties. Beyond identifying ways we can be influenced, I list out three techniques to help us make better decisions.

The War to Influence Engineers

This is just one example in a larger war that rages in this global era of computing: that for the hearts of engineers.

Some of the many groups that will try to shape our views include:
- Developer tools teams who want to sell their technology or grow their platform
- Companies who wants to hire us
- Open source developers who want to recruit contributors and users
- Venture capitalists who want to fund us
- Training programs that want to sell their courses
- Consulting firms that want us to hire them
- Social networks that are trying to maximize engagement, readership, and revenue
The content they create and sponsor (press releases, meetups, blog posts, conferences, speaker talks) play a critical role in the decisions we make — from the technology we adopt to the companies we join.

A press release indicating large traction (often with many missing numbers) may be a primary reason to join a startup. A blog post or popular talk on the latest database technology — like NoSQL — may be a reason to experiment and later switch. A college lecturer may partly be there to pitch their own company’s technology. The most upvoted Hacker News (HN) posts over a year may influence what technologies we choose to adopt. And yet, as consumers of this information, many of us don’t always critically examine the information and its source.

In startups, bad technology choices made based on these inputs can divert the focus, putting the mission at risk. For an individual engineer, learning the wrong technology can waste our time. Joining the wrong company that sells us false promises can set our career back.

Example: Recruiting engineers and raising venture capital in startups

A friend in VC once laughed after seeing an article extolling how well a startup was doing. After having been pitched by that company, he knew they were struggling and the article was their way to attract investor interest and continue hiring good talent before their funding ran out. I know many people who would have taken the exact opposite impression from the post. Crucially, with the right training, it would have been possible to suss out some of the reality behind this article.³

In a public example, Theranos popularized the idea of the pinprick blood test and had potentially huge social benefits. It must have been exciting for technical talent to join the company.

Yet many were attracted by universally adulatory press (example) that had been shaped by a motivated management team and media organizations that uncritically covered them.⁴ While much has been written about investor losses, Theranos’s exceptional engineers and scientists also likely lost important years to do truly impactful work and had their resumes sullied.
Example: Dev Tool Marketing at MongoDB

MongoDB was one of the fastest growing NoSQL databases in the late 2000s that promised a “modern” database experience.

In my deep dive into the early marketing strategy behind MongoDB, I pointed out issues relevant in dev tools marketing including:

Speakers invited to many MongoDB meetups/conferences would have their own motivations (trying to sell their consulting services, marketing for their own dev tools products)

Hack academies and early programmer blogs were valuable allies for pitching the MEAN stack, while not seeming like a sales pitch when seen on HN/Reddit

MongoDB would argue that nearly every use case should be solved with their database, without highlighting the tradeoffs in different use cases

College hackathons were a way to influence early developers who may not have fully understood the tradeoffs they were making
Have your own example? Email me at example at this domain. (my public key)

Even though I point out the issues in these case studies, I am far from a dispassionate scientist when on a mission for my team. Given that, it’s at least important to ensure that all engineers have the general tools to understand some of the messages we see. And in this era of fake news, selective facts, and troll armies, these lessons can also help us assess information far beyond engineering content.

Hidden Influences

Unlike previous generations of marketing, the external influences in engineering media today are often disguised.

The purpose of a press release or a sales call is clear. When this content is used to give a talk at a conference or inspire someone else to write a blog post, they blur the line. With a powerful marketing budget or fundraise, it becomes increasingly easy to publicize and convince others with one’s message, regardless of the veracity.⁵

I’ve had a personal experience with this in a different part of the world. After my engineering studies, I worked for the World Bank in Dili, East Timor. One day, we were explaining to a group of journalists⁶ the benefits of a project we were funding. There were many debates to be had for a thoughtful Timorese citizen⁷, but our focus was on how the World Bank perceived it. We predictably sent the journalists home with a press release extolling the project’s virtues.

The next day, our press release appeared verbatim in a local Timorese newspaper with no mention that it was a press release — under the author’s byline. It made sure that the World Bank’s key points were shared. And yet it raised troubling questions for me about how Timorese citizens used media to make decisions.⁸

Only later would I realize how prevalent this is around the world. In Silicon Valley, well placed friends tell me that they shudder at the information sources available to most — and how uncritically different messages are internalized. I worry especially about junior engineers (such as those in college and on r/learnprogramming) and those far from top tech centers, who may not realize how “the sausage is made.”

Example: Microservices hype

The engineering community has gone through large debates about the benefits of microservices. My own view is that the popularity of microservices in startups was a function of the hype and marketing spend around containerization. For example, compare a blog post on microservices at startups to Martin Fowler’s more nuanced view.

Many of these messages come from other engineers, open source teams, or training programs who are often trusted in a way few others are. This highlights a common mistake in startup engineering, where products/techniques pitched at mid-size company CTOs appear in pre-product market fit startups regardless of their appropriateness.

Example: Developer training programs

Training programs aren’t immune from the desire to engage readers and maximize the distribution for their content as they seek to increase readership and attract students. Some of the teachers they work with have their own motivations to push a certain technology — and their background may not be vetted.

Free Code Camp, a popular developer blog with 350k followers, carried a recent post that argued REST is dead and GraphQL is the future (“REST in Peace. Long Live GraphQL”). The article had been posted by the head of Free Code Camp and written by an author who was concurrently selling a GraphQL training program (my critique here).

Reflecting back on my own early days, I worry about junior engineers who might feel like they have to learn GraphQL based on an article like this rather than realizing all the conflicts involved. An older example from them is “The Real Reason to learn the MEAN stack: Employability” — which many of my most thoughtful friends would have strenuously disagreed with. I especially worry about what content like this does in early developer communities, where there many not be enough thoughtful engineers to disagree with these viewpoints.

Example: Content marketing in tech

Beyond direct influence, a surprising percentage of posts on engineering social media are content marketing. These posts aim to add value — but are written for the primary purpose of helping their companies sell their product. Readers can sometimes internalize them as “journalism,” even though their objective is to make a sale, influence a mindset shift, or improve SEO.

Union Square’s Fred Wilson notes how pervasive content marketing is in tech and encourages critical assessment of the message (emphasis added):

So how should entrepreneurs use this knowledge that is being imparted by VCs [like myself] …? Well first and foremost, you should see it as content marketing…

That doesn’t mean it isn’t useful or insightful. It may well be. But you should understand the business model supporting all of this free content. It is being generated to get you to come visit that VC and offer them to participate in your Seed or Series A round. That blog post that Joe claimed is not scripture in his tweet is actually an advertisement. Kind of the opposite of scripture, right?

For all these reasons and more, my most thoughtful friends use their own networks to understand reality instead: college/graduate school friends from select universities, frank talks with portfolio CEOs (for investors), ‘off the record’ conversations with connections at the various platform providers, trenchant blogs from front line engineers. Their information sources are less influenced by self-interest, providing a more objective view of the world. People can also say things privately they wouldn’t feel comfortable sharing broadly.

These friends of mine also have a deep-seated desire (specifically, a financial or technical motivation) to get at the truth, and so aim to understand the incentives driving every one of their sources. Though they may be outwardly agreeable, internally they’re deeply critical thinkers.

How Engineers Can Make Better Choices

These issues have a basic lesson: nothing is a replacement for surrounding oneself with primary sources, a community interested in the truth, and thoughtful engineers.

I’ve created a Github repo listing a few of the oft used inputs for startup engineering decisions that I’ve seen over my own startup journey. For each, I’ve noted how they work — and how they can be exploited. Contributions are welcome.

Beyond the self-interested factors, engineering media can be distorted for other reasons. For example, each upvote on Hacker News is equal — meaning that the world’s most thoughtful expert on a topic has the same voting power as someone inexperienced. Nearly every social media algorithm favors engagement over any other metric, meaning that what we read is simply a function of what those around us want to read (this echoes my research on what types of deaths are covered in a leading US newspaper).

Example: Cryptocurrency tribes and social media

In social media, the group you surround yourself by can shape your attitudes, even though engineering decisions should be based on empiricism and facts. Social media groups and algorithms pick content that confirms its users’ beliefs — and paint the opposite side in the worst light. This tribalism can be regularly seen in cryptocurrency subreddits, meaning that readers consume a distorted reality (see also filter bubbles).

As the cofounder of Coinbase, Fred Ehrsam, explains:

Cryptocurrencies create strong tribalism. Once you own a currency, your incentives are to make that currency go up in value. Crypto tribalism can be seen on reddit every day. Subreddits generate and report news that support their holdings. Crypto tribalism plays out in two common ways: 1) people promoting their own currency and 2) people discrediting other currencies. People promoting their own currency is evidenced by the imbalance of positive to negative news about a currency on its own subreddit.

For now, I’ll suggest three simple points that go far in assessing engineering media:

1. Identify incentives behind content that informs important decisions

First, consider a writer/speaker’s motivation — both directly (an employee or investor in the company) or indirectly (a public speaker who is a consultant looking for a gig or employee looking for a job/advisorship).

When I read a post or hear a speaker, I often ask myself a few questions:
- Background: What is their technical background and why are they well suited to speak on this topic?
- Objectives: What are their personal objectives in speaking/blogging? (e.g., recruiting, personal brand, new business); there almost always is one
- Conflicts: What conflicts might they have and how does this influence the message?
- Allies: Who else wants this content out there?
- Distributor Incentives; Why is the distributor (social network, media publication) putting this content in front of me? (who are the upvoters, what is the business model, etc.)
When we see a lot of press from a company, it’s often a sign that they want something from us. Their employees have consciously spent the time to sit down with a journalist or taken the time to write a post.⁹

2. Debunk claims

Second, inculcate a desire to debunk claims.

For example, companies regularly trumpet their (tailor-made) benchmarks that show their products in the best light. It’s often not that difficult to run our own benchmarks that reflect our real world use cases.

The direct reading of codebases is a powerful corrective (where available). It’s a primary source that doesn’t have ‘spin’ built in.

Looking back on trends in technology and where they ended up is another approach, which is akin to backtesting¹⁰: it’s easier to tune out hype, when we think about how people in a few years may perceive it, based on things we’ve seen in the past.

Say something in online communities when you disagree or see egregious incentive issues (here’s one favorite recent example on AV1 vs HEVC). On HN, this could be comments or a post of your own, even if it’s with a throwaway account. New social network designs might let us award karma to people who list out the key incentive issues in the future.

3. Build a community of respected peers and thoughtful experts

Third, examine the community that you lean on to determine important technical decisions.

It’s important to have a community of people who benefit from the truth, think critically, and have content expertise. This could be former employees, classmates, or your coworkers. You can augment this with the most thoughtful engineers you can find. Online, you can often find some of these groups in the comments of niche HN posts and open source mailing lists/chat rooms. In your early years, finding a thoughtful mentor is critical.

In many ways, this approach mirrors the web of trust in computer security, where we look to trusted friends — and people these trusted friends are connected to.

This point underlines the benefits of a thoughtful engineering curriculum taught by dispassionate teachers. I’m unaware of any school who teaches early engineers how they should best consume engineering media — or how to choose a technical tool. Often, this means that we’re then vulnerable for a few years, until we’ve incorporated the lessons from working on a good team.

Having a few thoughtful experts in our community is also critical when assessing deeply technical decisions like what database to use or how best to architect a system. The danger with some experts — the non-thoughtful variety — is that their confidence in their abilities means they don’t question many deep seated assumptions and can’t make adjustments when the world changes. Thoughtful experts have both content expertise and strong opinions that are weakly held.

Coda

We engineers share some similarities with the machine learning algorithms we design. In both, data is used to make inferences about future decisions. As such, it’s important to have a thoughtfully curated training data set — and to adjust for any errors when our engineering inputs are compromised.

I’ve hardly touched on all the other questions we should be discussing, beyond encouraging critical thinking and finding trusted networks of thoughtful engineers. For example, should we debate alternate Hacker News and Twitter algorithms with the hope of creating better engineers? Should there be conflict disclosures for open source developers? Should popular engineering blogs such as Free Code Camp have a board of engineers who technically assess content and the background of writers? Should we teach engineering media literacy in bootcamps and CS programs?

Still, I’m encouraged by the fact that critical thinking and testing hypothesis is such a natural part of how engineers and scientists approach the world. This mindset has a critical role to play far beyond the day to day engineering problems we aim to solve.

See the Hacking Engineers and Engineering Media Github. Contributions are welcome.
每天推荐一个 GitHub 优质开源项目和一篇精选英文科技或编程文章原文，欢迎关注开源日报。交流QQ群：202790710；电报群 https://t.me/OpeningSourceOrg

标签： 编程资源

2018年4月1日：开源日报第24期

什么是 awesome 项目

创建一个awesome 项目（列表）

Ten Machine Learning Algorithms You Should Know to Become a Data Scientist

1. Principal Component Analysis(PCA)/SVD

Libraries:

Introductory Tutorial:

2a. Least Squares and Polynomial Fitting

Libraries:

Introductory Tutorial:

2b. Constrained Linear Regression

Libraries:

Introductory Tutorial(s):

3. K means Clustering

Library:

Introductory Tutorial(s):

4. Logistic Regression

Library:

Introductory Tutorial(s):

5. SVM (Support Vector Machines)

Library:

Introductory Tutorial(s):

6. Feedforward Neural Networks

Libraries:

Introductory Tutorial(s):

7. Convolutional Neural Networks (Convnets)

Libraries:

Introductory Tutorial(s):

8. Recurrent Neural Networks (RNNs):

Library:

Introductory Tutorial(s):

9. Conditional Random Fields (CRFs)

Library:

Introductory Tutorial(s):

10. Decision Trees

Libraries

Introductory Tutorial:

TD Algorithms (Good To Have)

Libraries:

Introductory Tutorial(s):

2018年3月27日：开源日报第19期

Flow的优势

关于glow

关于作者

The War to Influence Engineers

Example: Recruiting engineers and raising venture capital in startups

Example: Dev Tool Marketing at MongoDB

Hidden Influences

Example: Microservices hype

Example: Developer training programs

Example: Content marketing in tech

How Engineers Can Make Better Choices

Example: Cryptocurrency tribes and social media

1. Identify incentives behind content that informs important decisions

2. Debunk claims

3. Build a community of respected peers and thoughtful experts

Coda

标签：编程资源