lexical analysis More...
Inheritance diagram for detail::lexer< BasicJsonType, InputAdapterType >:
Collaboration diagram for detail::lexer< BasicJsonType, InputAdapterType >:Public Types | |
| using | token_type = typename lexer_base< BasicJsonType >::token_type |
Public Types inherited from detail::lexer_base< BasicJsonType > | |
| enum class | token_type { uninitialized , literal_true , literal_false , literal_null , value_string , value_unsigned , value_integer , value_float , begin_array , begin_object , end_array , end_object , name_separator , value_separator , parse_error , end_of_input , literal_or_value } |
| token types for the parser More... | |
Public Member Functions | |
| lexer (const lexer &)=delete | |
| lexer (InputAdapterType &&adapter, bool ignore_comments_=false) noexcept | |
| lexer (lexer &&)=default | |
| ~lexer ()=default | |
| JSON_HEDLEY_RETURNS_NON_NULL constexpr const char * | get_error_message () const noexcept |
| return syntax error message | |
| constexpr number_float_t | get_number_float () const noexcept |
| return floating-point value | |
| constexpr number_integer_t | get_number_integer () const noexcept |
| return integer value | |
| constexpr number_unsigned_t | get_number_unsigned () const noexcept |
| return unsigned integer value | |
| constexpr position_t | get_position () const noexcept |
| return position of last read token | |
| string_t & | get_string () |
| return current string value (implicitly resets the token; useful only once) | |
| std::string | get_token_string () const |
| return the last read token (for errors only). | |
| lexer & | operator= (lexer &&)=default |
| lexer & | operator= (lexer &)=delete |
| token_type | scan () |
| bool | skip_bom () |
| skip the UTF-8 byte order mark | |
| void | skip_whitespace () |
Private Types | |
| using | char_int_type = typename char_traits< char_type >::int_type |
| using | char_type = typename InputAdapterType::char_type |
| using | number_float_t = typename BasicJsonType::number_float_t |
| using | number_integer_t = typename BasicJsonType::number_integer_t |
| using | number_unsigned_t = typename BasicJsonType::number_unsigned_t |
| using | string_t = typename BasicJsonType::string_t |
Private Member Functions | |
| void | add (char_int_type c) |
| add a character to token_buffer | |
| char_int_type | get () |
| int | get_codepoint () |
get codepoint from 4 hex characters following \u | |
| bool | next_byte_in_range (std::initializer_list< char_int_type > ranges) |
| check if the next byte(s) are inside a given range | |
| void | reset () noexcept |
| reset token_buffer; current character is beginning of token | |
| bool | scan_comment () |
| scan a comment | |
| token_type | scan_literal (const char_type *literal_text, const std::size_t length, token_type return_type) |
| token_type | scan_number () |
| scan a number literal | |
| token_type | scan_string () |
| scan a string literal | |
| void | unget () |
| unget current character (read it again on next get) | |
Static Private Member Functions | |
| static JSON_HEDLEY_PURE char | get_decimal_point () noexcept |
| return the locale-dependent decimal point | |
| static void | strtof (double &f, const char *str, char **endptr) noexcept |
| static void | strtof (float &f, const char *str, char **endptr) noexcept |
| static void | strtof (long double &f, const char *str, char **endptr) noexcept |
Private Attributes | |
| char_int_type | current = char_traits<char_type>::eof() |
| the current character | |
| const char_int_type | decimal_point_char = '.' |
| the decimal point | |
| std::size_t | decimal_point_position = std::string::npos |
| the position of the decimal point in the input | |
| const char * | error_message = "" |
| a description of occurred lexer errors | |
| InputAdapterType | ia |
| input adapter | |
| const bool | ignore_comments = false |
| whether comments should be ignored (true) or signaled as errors (false) | |
| bool | next_unget = false |
| whether the next get() call should just return current | |
| position_t | position {} |
| the start position of the current token | |
| string_t | token_buffer {} |
| buffer for variable-length tokens (numbers, strings) | |
| std::vector< char_type > | token_string {} |
| raw input token string (for error messages) | |
| number_float_t | value_float = 0 |
| number_integer_t | value_integer = 0 |
| number_unsigned_t | value_unsigned = 0 |
Additional Inherited Members | |
Static Public Member Functions inherited from detail::lexer_base< BasicJsonType > | |
| JSON_HEDLEY_RETURNS_NON_NULL static JSON_HEDLEY_CONST const char * | token_type_name (const token_type t) noexcept |
| return name of values of type token_type (only used for errors) | |
lexical analysis
This class organizes the lexical analysis during JSON deserialization.
|
private |
|
private |
|
private |
|
private |
|
private |
|
private |
| using detail::lexer< BasicJsonType, InputAdapterType >::token_type = typename lexer_base<BasicJsonType>::token_type |
|
inlineexplicitnoexcept |
|
delete |
|
default |
|
default |
|
inlineprivate |
add a character to token_buffer
Definition at line 8475 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::token_buffer.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::next_byte_in_range(), detail::lexer< BasicJsonType, InputAdapterType >::scan_number(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_string().
Here is the caller graph for this function:
|
inlineprivate |
Definition at line 8411 of file json.h.
References detail::position_t::chars_read_current_line, detail::position_t::chars_read_total, detail::lexer< BasicJsonType, InputAdapterType >::current, detail::lexer< BasicJsonType, InputAdapterType >::ia, JSON_HEDLEY_LIKELY, detail::position_t::lines_read, detail::lexer< BasicJsonType, InputAdapterType >::next_unget, detail::lexer< BasicJsonType, InputAdapterType >::position, and detail::lexer< BasicJsonType, InputAdapterType >::token_string.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get_codepoint(), detail::lexer< BasicJsonType, InputAdapterType >::next_byte_in_range(), detail::lexer< BasicJsonType, InputAdapterType >::scan_comment(), detail::lexer< BasicJsonType, InputAdapterType >::scan_literal(), detail::lexer< BasicJsonType, InputAdapterType >::scan_number(), detail::lexer< BasicJsonType, InputAdapterType >::scan_string(), detail::lexer< BasicJsonType, InputAdapterType >::skip_bom(), and detail::lexer< BasicJsonType, InputAdapterType >::skip_whitespace().
Here is the caller graph for this function:
|
inlineprivate |
get codepoint from 4 hex characters following \u
For input "\u c1 c2 c3 c4" the codepoint is: (c1 * 0x1000) + (c2 * 0x0100) + (c3 * 0x0010) + c4 = (c1 << 12) + (c2 << 8) + (c3 << 4) + (c4 << 0)
Furthermore, the possible characters '0'..'9', 'A'..'F', and 'a'..'f' must be converted to the integers 0x0..0x9, 0xA..0xF, 0xA..0xF, resp. The conversion is done by subtracting the offset (0x30, 0x37, and 0x57) between the ASCII value of the character and the desired integer value.
Definition at line 7240 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::current, detail::lexer< BasicJsonType, InputAdapterType >::get(), and JSON_ASSERT.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan_string().
Here is the call graph for this function:
Here is the caller graph for this function:
|
inlinestaticprivatenoexcept |
return the locale-dependent decimal point
Definition at line 7214 of file json.h.
References JSON_ASSERT.
|
inlineconstexprnoexcept |
return syntax error message
Definition at line 8552 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::error_message.
|
inlineconstexprnoexcept |
return floating-point value
Definition at line 8498 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::value_float.
|
inlineconstexprnoexcept |
return integer value
Definition at line 8486 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::value_integer.
|
inlineconstexprnoexcept |
return unsigned integer value
Definition at line 8492 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::value_unsigned.
|
inlineconstexprnoexcept |
return position of last read token
Definition at line 8519 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::position.
Referenced by visualizer.core.Node::_update_position().
Here is the caller graph for this function:
|
inline |
return current string value (implicitly resets the token; useful only once)
Definition at line 8504 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::decimal_point_char, detail::lexer< BasicJsonType, InputAdapterType >::decimal_point_position, and detail::lexer< BasicJsonType, InputAdapterType >::token_buffer.
|
inline |
return the last read token (for errors only).
Will never contain EOF (an arbitrary value that is not a valid char value, often -1), because 255 may legitimately occur. May contain NUL, which should be escaped.
Definition at line 8527 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::token_string.
|
inlineprivate |
check if the next byte(s) are inside a given range
Adds the current byte and, for each passed range, reads a new byte and checks if it is inside the range. If a violation was detected, set up an error message and return false. Otherwise, return true.
| [in] | ranges | list of integers; interpreted as list of pairs of inclusive lower and upper bound, respectively |
Definition at line 7288 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::add(), detail::lexer< BasicJsonType, InputAdapterType >::current, detail::lexer< BasicJsonType, InputAdapterType >::error_message, detail::lexer< BasicJsonType, InputAdapterType >::get(), JSON_ASSERT, and JSON_HEDLEY_LIKELY.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan_string().
Here is the call graph for this function:
Here is the caller graph for this function:
|
default |
|
delete |
|
inlineprivatenoexcept |
reset token_buffer; current character is beginning of token
Definition at line 8393 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::current, detail::lexer< BasicJsonType, InputAdapterType >::decimal_point_position, detail::lexer< BasicJsonType, InputAdapterType >::token_buffer, and detail::lexer< BasicJsonType, InputAdapterType >::token_string.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan_number(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_string().
Here is the caller graph for this function:
|
inline |
Definition at line 8588 of file json.h.
References detail::position_t::chars_read_total, detail::lexer< BasicJsonType, InputAdapterType >::current, detail::lexer< BasicJsonType, InputAdapterType >::error_message, detail::lexer< BasicJsonType, InputAdapterType >::ignore_comments, detail::lexer< BasicJsonType, InputAdapterType >::position, detail::lexer< BasicJsonType, InputAdapterType >::scan_comment(), detail::lexer< BasicJsonType, InputAdapterType >::scan_literal(), detail::lexer< BasicJsonType, InputAdapterType >::scan_number(), detail::lexer< BasicJsonType, InputAdapterType >::scan_string(), detail::lexer< BasicJsonType, InputAdapterType >::skip_bom(), and detail::lexer< BasicJsonType, InputAdapterType >::skip_whitespace().
Here is the call graph for this function:
|
inlineprivate |
scan a comment
Definition at line 7915 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::error_message, detail::lexer< BasicJsonType, InputAdapterType >::get(), and detail::lexer< BasicJsonType, InputAdapterType >::unget().
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan().
Here is the call graph for this function:
Here is the caller graph for this function:
|
inlineprivate |
| [in] | literal_text | the literal text to expect |
| [in] | length | the length of the passed literal text |
| [in] | return_type | the token type to return on success |
Definition at line 8373 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::current, detail::lexer< BasicJsonType, InputAdapterType >::error_message, detail::lexer< BasicJsonType, InputAdapterType >::get(), JSON_ASSERT, and JSON_HEDLEY_UNLIKELY.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan().
Here is the call graph for this function:
Here is the caller graph for this function:
|
inlineprivate |
scan a number literal
This function scans a string according to Sect. 6 of RFC 8259.
The function is realized with a deterministic finite state machine derived from the grammar described in RFC 8259. Starting in state "init", the input is read and used to determined the next state. Only state "done" accepts the number. State "error" is a trap state to model errors. In the table below, "anything" means any character but the ones listed before.
| state | 0 | 1-9 | e E | + | - | . | anything |
|---|---|---|---|---|---|---|---|
| init | zero | any1 | [error] | [error] | minus | [error] | [error] |
| minus | zero | any1 | [error] | [error] | [error] | [error] | [error] |
| zero | done | done | exponent | done | done | decimal1 | done |
| any1 | any1 | any1 | exponent | done | done | decimal1 | done |
| decimal1 | decimal2 | decimal2 | [error] | [error] | [error] | [error] | [error] |
| decimal2 | decimal2 | decimal2 | exponent | done | done | done | done |
| exponent | any2 | any2 | [error] | sign | sign | [error] | [error] |
| sign | any2 | any2 | [error] | [error] | [error] | [error] | [error] |
| any2 | any2 | any2 | done | done | done | done | done |
The state machine is realized with one label per state (prefixed with "scan_number_") and goto statements between them. The state machine contains cycles, but any cycle can be left when EOF is read. Therefore, the function is guaranteed to terminate.
During scanning, the read bytes are stored in token_buffer. This string is then converted to a signed integer, an unsigned integer, or a floating-point number.
. to work with the locale-dependent converters. Definition at line 8040 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::add(), detail::lexer< BasicJsonType, InputAdapterType >::current, detail::lexer< BasicJsonType, InputAdapterType >::decimal_point_char, detail::lexer< BasicJsonType, InputAdapterType >::decimal_point_position, detail::lexer< BasicJsonType, InputAdapterType >::error_message, detail::lexer< BasicJsonType, InputAdapterType >::get(), JSON_ASSERT, detail::lexer< BasicJsonType, InputAdapterType >::reset(), detail::lexer< BasicJsonType, InputAdapterType >::strtof(), detail::lexer< BasicJsonType, InputAdapterType >::token_buffer, detail::lexer< BasicJsonType, InputAdapterType >::unget(), detail::lexer< BasicJsonType, InputAdapterType >::value_float, detail::lexer< BasicJsonType, InputAdapterType >::value_integer, and detail::lexer< BasicJsonType, InputAdapterType >::value_unsigned.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan().
Here is the call graph for this function:
Here is the caller graph for this function:
|
inlineprivate |
scan a string literal
This function scans a string according to Sect. 7 of RFC 8259. While scanning, bytes are escaped and copied into buffer token_buffer. Then the function returns successfully, token_buffer is not null-terminated (as it may contain \0 bytes), and token_buffer.size() is the number of bytes in the string.
Definition at line 7325 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::add(), detail::lexer< BasicJsonType, InputAdapterType >::current, detail::lexer< BasicJsonType, InputAdapterType >::error_message, detail::lexer< BasicJsonType, InputAdapterType >::get(), detail::lexer< BasicJsonType, InputAdapterType >::get_codepoint(), JSON_ASSERT, JSON_HEDLEY_LIKELY, JSON_HEDLEY_UNLIKELY, detail::lexer< BasicJsonType, InputAdapterType >::next_byte_in_range(), and detail::lexer< BasicJsonType, InputAdapterType >::reset().
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan().
Here is the call graph for this function:
Here is the caller graph for this function:
|
inline |
skip the UTF-8 byte order mark
Definition at line 8565 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::get(), and detail::lexer< BasicJsonType, InputAdapterType >::unget().
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan().
Here is the call graph for this function:
Here is the caller graph for this function:
|
inline |
Definition at line 8579 of file json.h.
References detail::lexer< BasicJsonType, InputAdapterType >::current, and detail::lexer< BasicJsonType, InputAdapterType >::get().
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan().
Here is the call graph for this function:
Here is the caller graph for this function:
|
inlinestaticprivatenoexcept |
|
inlinestaticprivatenoexcept |
Definition at line 7983 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan_number().
Here is the caller graph for this function:
|
inlinestaticprivatenoexcept |
|
inlineprivate |
unget current character (read it again on next get)
We implement unget by setting variable next_unget to true. The input is not changed - we just simulate ungetting by modifying chars_read_total, chars_read_current_line, and token_string. The next call to get() will behave as if the unget character is read again.
Definition at line 8448 of file json.h.
References detail::position_t::chars_read_current_line, detail::position_t::chars_read_total, detail::lexer< BasicJsonType, InputAdapterType >::current, JSON_ASSERT, JSON_HEDLEY_LIKELY, detail::position_t::lines_read, detail::lexer< BasicJsonType, InputAdapterType >::next_unget, detail::lexer< BasicJsonType, InputAdapterType >::position, and detail::lexer< BasicJsonType, InputAdapterType >::token_string.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan_comment(), detail::lexer< BasicJsonType, InputAdapterType >::scan_number(), and detail::lexer< BasicJsonType, InputAdapterType >::skip_bom().
Here is the caller graph for this function:
|
private |
the current character
Definition at line 8684 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get(), detail::lexer< BasicJsonType, InputAdapterType >::get_codepoint(), detail::lexer< BasicJsonType, InputAdapterType >::next_byte_in_range(), detail::lexer< BasicJsonType, InputAdapterType >::reset(), detail::lexer< BasicJsonType, InputAdapterType >::scan(), detail::lexer< BasicJsonType, InputAdapterType >::scan_literal(), detail::lexer< BasicJsonType, InputAdapterType >::scan_number(), detail::lexer< BasicJsonType, InputAdapterType >::scan_string(), detail::lexer< BasicJsonType, InputAdapterType >::skip_whitespace(), and detail::lexer< BasicJsonType, InputAdapterType >::unget().
|
private |
the decimal point
Definition at line 8707 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get_string(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_number().
|
private |
the position of the decimal point in the input
Definition at line 8709 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get_string(), detail::lexer< BasicJsonType, InputAdapterType >::reset(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_number().
|
private |
a description of occurred lexer errors
Definition at line 8699 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get_error_message(), detail::lexer< BasicJsonType, InputAdapterType >::next_byte_in_range(), detail::lexer< BasicJsonType, InputAdapterType >::scan(), detail::lexer< BasicJsonType, InputAdapterType >::scan_comment(), detail::lexer< BasicJsonType, InputAdapterType >::scan_literal(), detail::lexer< BasicJsonType, InputAdapterType >::scan_number(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_string().
|
private |
input adapter
Definition at line 8678 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get().
|
private |
whether comments should be ignored (true) or signaled as errors (false)
Definition at line 8681 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::scan().
|
private |
whether the next get() call should just return current
Definition at line 8687 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get(), and detail::lexer< BasicJsonType, InputAdapterType >::unget().
|
private |
the start position of the current token
Definition at line 8690 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get(), detail::lexer< BasicJsonType, InputAdapterType >::get_position(), detail::lexer< BasicJsonType, InputAdapterType >::scan(), and detail::lexer< BasicJsonType, InputAdapterType >::unget().
|
private |
buffer for variable-length tokens (numbers, strings)
Definition at line 8696 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::add(), detail::lexer< BasicJsonType, InputAdapterType >::get_string(), detail::lexer< BasicJsonType, InputAdapterType >::reset(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_number().
|
private |
raw input token string (for error messages)
Definition at line 8693 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get(), detail::lexer< BasicJsonType, InputAdapterType >::get_token_string(), detail::lexer< BasicJsonType, InputAdapterType >::reset(), and detail::lexer< BasicJsonType, InputAdapterType >::unget().
|
private |
Definition at line 8704 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get_number_float(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_number().
|
private |
Definition at line 8702 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get_number_integer(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_number().
|
private |
Definition at line 8703 of file json.h.
Referenced by detail::lexer< BasicJsonType, InputAdapterType >::get_number_unsigned(), and detail::lexer< BasicJsonType, InputAdapterType >::scan_number().