In this tutorial, we will understand tokens in Java in simple words.
A Java program is basically made up of a group of classes and methods. A class is a container that contains a set of declaration statements and methods containing executable statements.
A statement consists of variables, constants, operators, keywords, comments, identifiers, punctuators, etc. When you run the program, comments are ignored by the Java compiler and executable statements are executed. Comments are not part of executable statements.
A Java source file (.java file) is called a compilation unit. It consists of a sequence of tokens, comments, and whitespace. The Java compiler ignores comments, and whitespaces. It only process tokens. So, let us understand what tokens are?
What are Tokens in Java?
Tokens are the various elements in the Java program that are identified by Java compiler. A token is the smallest individual element (unit) in a program that is meaningful to the compiler.
In simple words, a Java program is a group of tokens, comments, and white spaces. Consider the following examples below.
Example 1:
final double p = 3.14; // A constant.This statement consists of six tokens: “final”, “double”, “p”, “=”, “3.14”, and “;”.
Example 2:
x = a + b; // An expression.Similarly, this statement consists of six tokens: “x”, “=”, “a”, “+”, “b”, and “;”.
Example 3:
v = Math.pow(10, 1); // An inbuilt java function.This statement consists of total 11 tokens: “v”, “=”, “Math”, “.”, “pow”, “(“, “10”, “,”, “1”, “)” and “;”.
Types of Tokens
Java language contains five types of tokens that are as follows:
- Keywords (Reserved Keywords)
- Identifiers
- Literals
- Operators
- Separators
We will understand about them one by one in the further tutorial.
Key Points About Tokens
- Tokens are the basic building blocks of a Java program, including identifiers, keywords, literals, operators, and separators.
- Keywords represent reserved words predefined by the Java language. For example, class, public, static, if, else, etc. You cannot use these words as identifiers for the names of variables, methods, classes, etc.
- Identifiers represent names given to elements, such as variables, methods, classes, and interfaces. Identifiers must start with a letter (A-Z or a-z), underscore (_), or dollar sign ($), followed by any combination of letters, digits, underscores, or dollar signs.
- Literals are constants or fixed values in the program that don’t change during the execution of the program. They include integer, floating-point, character, string, and boolean literals.
- Operators are symbols in Java or other programming languages that perform various types of operations variables and values. Examples of operators are arithmetic operators (+, -, *, /), comparison operators (<, >, ==, !=), logical operators (&&, ||), etc.
- Separators (also known as delimiters) are characters that separate elements within the program code. Examples of separators are parentheses (( )), braces ({ }), brackets ([ ]), semicolon (;), and comma (,).
- The process of breaking down the source code into valid tokens is known as tokenization.
- Whitespace is used to separate tokens in Java but does not affect the execution of the program. It includes spaces, tabs, and newline characters.
- Java is case-sensitive, meaning it distinguishes between uppercase and lowercase letters. For example, myCollege and MyCollege are different identifiers.
