Basics of Artificial Intelligence – III

Some artificial intelligence algorithms like input values to be normalized. This means that all data is presented within a predefined range, typically either 0 to 1 or -1 to 1. Normalization algorithms take an array of input values and return an array of normalized values.

Denormalization is the opposite process. In denormalization, an input array of normalized values is presented and the original values are returned. Denormalization is useful when the output value of an AI algorithm is normalized. Since the normalized value is not in an expected range, the user must denormalize to determine the real number.

A simple example of number normalization is the Celsius temperature scale. All temperatures where water exists as a liquid exist between the values of 0 and 100. To normalize the temperature for an AI algorithm, I could simply divide each input by 100 to create an array of numbers between 0 and 1. When the output value is .17, the user would denormalize by multiplying by 100 to get a value of 17 degrees.

Of course, most normalization is not this simple, so we use algorithms to do the work.

public static float[] normalizeData(final float[] inputVector, final float minVal, final float maxVal) {
	float[] normalizedData = new float[inputVector.length];
	float dataRange = maxVal - minVal;
	for(int i = 0; i < inputVector.length; ++i) {
		float d = inputVector[i] - minVal;
		float percent = d / dataRange;
		float dnorm = NORMALIZE_RANGE * percent;
		float norm = NORMALIZE_LOW_VALUE + dnorm;
		normalizedData[i] = norm;
	}
	return normalizedData;
}

Note that two constants are defined outside this function. The NORMALIZE_RANGE which is 2 when normalizing to the range of -1 to 1 and the NORMALIZE_RANGE is 1 if we are normalizing to a range of 0 to 1. Additionally, the NORMALIZE_LOW_VALUE is the low value for normalization, either -1 or 0.

In the above normalization function, the user provides an array of input values as well as a min and max value for normalization. Then, we create a new array to hold the normalized values. The code then iterates through each input value and creates the normalized value to add to the normalized data array to return to the user. The actual normalization takes the following steps:

  • subtract the minimum value from the input value
  • divide the output by the data range to determine a percentage
  • multiple the normalized range by the percent
  • Add the value to the normalized low value.

For a concrete example, consider normalizing degrees Fahrenheit. If we were to input an array of daily temperates, we might have [70, 75, 68]. For the normalization range, we would pick 32 and 212. Following the above steps for the first temperature:

  • 70 – 32 = 38
  • 38 / (212 – 32) = .21
  • 2 * .21 = .42
  • -1 + .42 = -.58

If we followed through with the other temperatures, we would end with an output array of [-.58, -.52, -.60]. To denormalize, the below denormalization function can be used. Note, you must use the same min and max values that you used in normalization or your denormalized output value will not be the same scale as your input values!

public static float[] denormalizeData(final float[] normalizedData, final float minVal, final float maxVal) {
	float[] denormalizedData = new float[normalizedData.length];
	float dataRange = maxVal - minVal;
	for(int i = 0; i < normalizedData.length; ++i) {
		float dist = normalizedData[i] - NORMALIZE_LOW_VALUE;
		float pct = dist / NORMALIZE_RANGE;
		float dnorm = pct * dataRange;
		denormalizedData[i] = dnorm + minVal;
	}
	return denormalizedData;
}

This is the most basic normalization function. Other options may be to use the reciprocal of a number (but this only works for number greater than 1 or less than -1) or to use a Z-score.

Basics of Artificial Intelligence – II

Last week, we talked about distance calculations for Artificial Intelligence. Once you’ve learned how to calculate distance, you need to learn how to calculate an overall error for your algorithm. There are three main algorithms for error calculation. Sum of Squares, Mean Squared, and Root Mean Squared. They are all relatively simple, but are key to any Machine Learning algorithm. As an AI algorithm iterates over data time and time again, it will try to find a better solution than the previous iteration. A lower error score indicates a better answer and progress toward the best solution.

The error algorithms are similar to the distance algorithms. However, distance measures how far apart two points are whereas error measures how far the AI output answers are from the expected answers. The three algorithms below show how each error is calculated. Note that each one builds on the one before it. The sum of squares error is – as the name suggests – a summation of the square of the errors of each answer. Note that as the number of answers increases, the sum of squares value will too. Thus, to compare errors with different numbers of values, we need to divide by the number of items to get the mean squared error. Finally, if you want to have a number in a similar range to the original answer, you need to take the square root of the mean squared error.

public static float sumOfSquares(final float[] expected, final float[] actual) {
	float sum = 0;
	for(int i = 0; i < expected.length; ++i) {
		sum += Math.pow(expected[i] - actual[i], 2);
	}
	return sum;
}
	
public static float meanSquared(final float[] expected, final float[] actual) {
	return sumOfSquares(expected, actual)/expected.length;
}
	

public static float rootMeanSquared(final float[] expected, final float[] actual) {
	return (float)Math.sqrt(meanSquared(expected,actual));
}

Basics of Artificial Intelligence – I

For the next several weeks, I’m going to write about some basics of artificial intelligence. AI has been around for decades, but has become particularly popular during the last 20 years thanks to advances in computing. In short, artificial intelligence aims to use computers to solve complex problems quicker and more accurately than human can. Early AI was far different than what we have today. Typically, early AI systems would use complex logic programmed into the system to solve problems. Examples include Dijkstra’s Algorithm or the logic programmed into most games. Modern systems, however, are capable of actually learning for themselves given enough data.

Distance Algorithms

The first set of algorithms necessary to understand AI are distance algorithms. These algorithms are used to determine how close a system is to the right answer. This is necessary when an AI system is learning so that it knows how far off the answer it is. The three main distance algorithms are Euclidian, Manhattan, and Chebyshev. Euclidian distance measures distance as a straight line “as the crow flies” between points on a grid. Manhattan distance travels along one axis and then another, like a taxi traversing New York City. Finally, Chebyshev distance travels like a King on a chessboard alternating between each axis as it gets closer to the target.

In each of the code snippets below, written in Java, two vectors are passed in – v1 and v2 – where each vector represents a data point. In each instance, the size of the vector would determine the dimensionality of the data. For example, a float[2] would be a 2-D vector which could be plotted on a cartesian plot.

Euclidian Distance Algorithm

public static float euclidean(final float[] v1, final float[] v2) {
	float sum = 0;
	for(int i = 0; i < v1.length; ++i) {
		sum += (v1[i] - v2[i]) * (v1[i] - v2[i]);
	}
	return (float)Math.sqrt(sum);
}

In the above code, we iterate through two arrays of floating point numbers and then sum the squares of the differences. Finally, return the square root to determine the distance.

Manhattan Distance Algorithm

public static float manhattan(final float[] v1, final float[] v2) {
	float sum = 0;
	for(int i = 0; i < v1.length; ++i) {
		sum += (float)Math.abs(v1[i] - v2[i]);
	}
	return sum;
}

For the Manhattan distance, we calculate and return the sum of the absolute values of the differences.

Chebyshev Distance Algorithm

public static float chebyshev(final float[] v1, final float[] v2) {
	float result = 0;
	for(int i = 0; i < v1.length; ++i) {
		float d = Math.abs(v1[i] - v2[i]);
		result = Math.max(d, result);
	}
	return result;
}

Finally, in the Chebyshev algorithm, the value is the maximum dimension in any direction.

Jupyter & Scikit-Learn for Artificial Intelligence

Jupiter

Many people are interested in artificial intelligence and machine learning, but what framework should you start out with to hit the ground running fast? Jupyter and Scikit-Learn!

Jupyter

Jupyter is a web-based development environment well-suited for Python development and artificial intelligence. What makes it great for AI development is the nature of the web platform. You can add code blocks, generate images and charts, or add markdown to document what you’re doing. What is also useful is that each block of code within the IDE can be executed as needed by clicking the run button at the top. This is something you will do over and over as you tune your AI parameters to get better accuracy.

The Jupiter Notebook is installed directly on your computer, so you can access local files. This is critical for being able to access the data you will use to train your AI model and ensures optimal performance when using large data sets.

Scikit-Learn

There are countless frameworks for AI, but Scikit-Learn is my favorite. Since it’s a Python-based framework, it’s relatively easy for anyone to work with. Furthermore, since Python is heavily used in artificial intelligence research, you’re using a tool well known to other practitioners in the field. Scikit-Learn supports just about anything you could want for AI including logistic regression, k-nearest neighbors, neural networks, naive bayes, and support vector machines. Even better, Scikit-learn makes it so simple to implement that you don’t really need to know much about the underlying technology.

Part of what makes scikit-learn so useful is how easy it is to create charts or graphs. This can include a typical confusion matrix or any plot for your data. This can be really powerful in AI to better understand the data and what relationships may exist.

Getting Started

If you’re interested in getting started, visit jupyter.org to download their IDE. Then, visit scikit-learn.org for installation instructions. If you’re familiar with Python, you’re ready to start with the scikit-learn tutorial on their website. If not, take a trip to YouTube and find some videos on Python development!

Skills All Developers Should Know

Tools

When I talk with developers, it’s amazing how many of them are disappointed with the limited job opportunities available to them. Often, the problem is that they lack many of the common skills required to be an effective programmer in today’s market. These developers have managed to become specialists in a very specific technology stack which isn’t widely used elsewhere. I’ll tell them some things they should learn, and – surprisingly – they’ll often argue that the specific skill I mentioned isn’t that useful. What are those skills?

HTML

First and foremost, every developer today needs to know HTML. In our web-based world, it’s hard to find work that doesn’t require HTML at some point. This is obviously true for web development, but is equally valuable with hybrid mobile technologies such as Cordova or Ionic. Or, maybe you need to create documentation for you application that will be accessed on the web.

JavaScript

While every developer may use a different programming language for backend development, we all use JavaScript for web programming. But JavaScript goes well beyond that. Today, JavaScript can be used for developing backend services with Node or even interfacing with Arduino hardware with the Johnny5 library.

Git

Developers need to be comfortable with Git from the command line. GUIs are nice, but when you need to automate a build script or download code from a command prompt on a remote computer, the Git command line is essential.

BASH

Like it or not, much of the cloud runs on Linux servers. As such, a knowledge of BASH is essential in today’s world. While many developers prefer a Windows only world, it’s just not the case when you have to deal with Google or AWS cloud services.

SQL

It’s amazing to me how many junior developers are unfamiliar with SQL. In today’s marketplace, nearly every application needs to interact with a database somewhere. Whether it’s an enterprise application using Oracle or an Android app using SQLite, SQL is the common way to interact with a database.

Conclusion

It may be noted that – other than JavaScript – no programming languages are on the list. Whether you develop in C, C++, Java, Swift or some other language, the above skills will apply. There are countless toolkits and frameworks, languages and environments. However, nearly all of them will require some mix of knowledge of HTML, JavaScript, Git, Bash, and SQL

Simple Text Search

Search

I often need to search a directory of code for instances of a specific word. I like tools that I can use from the command line so that I can execute them from an SSH session across the network. Here’s how I do it.

#!/bin/bash

SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

if [ $# -ne 1 ]
then
        echo Call is: `basename $0` string
else
        for file in `find . -type f | cut -c3-`
        do
                count=`cat "$file" | grep -i $1 | wc -l`
                if [ $count -gt 0 ]
                then
                        echo "******"$file"******"
                        cat "$file" | grep -i $1
                fi
        done
fi
IFS=$SAVEIFS

Line Counting

Codes

A common metric used in software development is lines of code. While it’s not always a useful metric, it’s sometimes nice to know just how large a project is. I have a script I use to count lines of code in a folder. It will iterate through a variety of common file extensions and count all lines of code excluding blank lines. It’s written using sh, so it works on Unix, Linux, or MacOS, and should work on Windows if you have the Bash subsystem installed.

#!/bin/bash

# add additional extensions here
extensions=(bas c cc cob cpp cs cshtml fth f90 go h html java js jsp m pas php pl py sc sh sql ts xhtml)

for extension in ${extensions[@]}
do
  lines=`find . -name "*.$extension" -type f -exec cat {} \; | tr -d '[:blank:]' 2> /dev/null | grep -v '^$' | wc -l `
  if [ $lines -ne 0 ]
  then 
    echo $extension: $lines
  fi
done

Languages vs Algorithms

Programming

I was recently discussing a project with a potential client. His project was using a language I haven’t used heavily in quite some time. He was disappointed because he was struggling to find developers to work on his project. I indicated that I would not have trouble getting back up to speed on the language and being able to move his project forward.

This really highlighted in my mind what really matters in development – it’s not languages, it’s algorithms and software design. All programming languages are, at their core, pretty similar. For example, all languages include:

  1. Syntax for creating variables,
  2. Selectively executing a block of code based on one or more conditions
  3. Repeating a portion of code multiple times
  4. Encapsulating code into functions

These basic structures are responsible for all the software you will ever use. Whether it’s a complex operating system like Windows or MacOS, a cell phone application on iOS or Android, or an application on the cloud, they all rely on the basic elements above.

Once you understand the basic syntax of a language, you move into the more important stuff – algorithms. Algorithms are a set of instructions to perform a given task. Regardless of the language, the algorithm will be largely the same. Next up on the hierarchy is software design. Knowing syntax and algorithms is great, but to really develop quality software you need to be able to design the interactions between components. None of these things vary substantially between languages. Languages are merely the syntactic rules you must follow to define your algorithms and implement your design.

I think people need to worry less about what language a developer uses and look for higher-order skills including algorithms and design. That’s what really defines the skill of a programmer!

Coding Style

Writing

One of the most important skills for any team of engineers is to develop a consistent coding style. Without it, development and debugging times increase and productivity decreases. Coding style is much like the rules of writing a story. Sentences start with a capital letter and end in a period. New ideas are started in new paragraphs. Spaces exist between letters of words. If someone were to write a book without following those conventions, it would be substantially more difficult to read. Coding style works the same way.

One of the projects I worked for years was a complicated telephony system. It included low-level code for interfacing with the phone boards as well as high-level code for displaying data to the user. This was a lengthy project that took substantial time to develop. During development, John and I worked to develop a consistent coding style. A mentor on the team, Mark, would review our code, provide feedback, and fanatically insist on standard coding conventions and style.

At first, this seemed silly to me. I’m a developer, let me do what I do! But, as time went on, it became increasingly apparent how valuable it was. After a while, I was unable to tell the code I wrote from what John wrote. While this may seem silly, this meant that when bugs were found, either of us could feel equally comfortable understanding and changing the code.

Today, when I learn a new language, one of the first things I try to do is learn standard coding conventions for that language. Some languages, like Java, have well established and followed conventions. Unfortunately, many other languages do not. When that happens, the team needs to determine what conventions they will follow for their team. If not, expect bugs to take longer and development to be slower.

Development Environments

Workbench

As a developer, it’s a great time to be alive. We have countless options for development languages and frameworks. But, sadly, many of them leave much to be desired in terms of tools and environments.

The Model Environment

As a veteran Java developer, I’ve been spoiled with excellent tools. We have numerous IDEs including Eclipse, NetBeans, and IntelliJ. Everyone has their preference, but all three are excellent. Likewise, we have numerous build and dependency management packages such as Maven and Gradle. We have test frameworks, automation frameworks, debuggers, profilers, and anything else you could want to engineer even the most complex of application. In short, Java provides the best development environment there is. Seriously. Few languages have IDEs as good as what Java has. Even fewer have tools that can complete with Maven or Gradle.

When Will Other Languages Catch Up?

I continue to see how Python is displacing other languages. But when will it catch up to Java in terms of development environment? Are there any IDEs that complete with Eclipse for Python? (NOTE: Eclipse can actually be used for Python, but it always seemed to me more of a hack than a real Python IDE) How about debuggers? Go ahead and try debugging Python or JavaScript code and you’ll quickly see how far behind Java they are. Not to mention dealing with libraries and environments!

A Call to Action

As we continue to shift to newer languages, I’m concerned that the next generation of developers will actually have a more difficult time than we do today. Why, as a development community, are we only focused on creating new languages but not on creating solid environments? How are Go, Python, Rust, and the myriad of other languages ever supposed to achieve the same widespread adoption of Java if we never had the rest of the tools we need to get the job done? I want to enjoy the benefits of some of the newer languages to come along. But, until they can complete with Java, I’m unable to stake the success of my customers on them.