How do I read a JSONL file in Go?

Use bufio.NewScanner to read the file line by line, then parse each line with json.Unmarshal into a struct or map. Alternatively, use json.NewDecoder for streaming decoding without explicit line splitting. Both approaches process one record at a time, keeping memory usage low.

What is the difference between bufio.Scanner and json.Decoder for JSONL?

bufio.Scanner reads raw lines and requires a separate json.Unmarshal call, giving you access to raw bytes and line numbers. json.Decoder reads directly from an io.Reader and decodes JSON values sequentially, which is more efficient but does not expose raw line data. Use Scanner when you need line-level error reporting, and Decoder when you want maximum throughput.

How do I write a JSONL file in Go?

Use json.Marshal to serialize each record, append a newline byte, and write to a file. For better performance, use json.NewEncoder wrapping a bufio.Writer. The encoder automatically adds a newline after each Encode call. Always call Flush on the buffered writer before closing the file.

How do I process JSONL concurrently in Go?

Use the fan-out pattern: read records in one goroutine, send them through a channel, and process them in multiple worker goroutines. Use sync.WaitGroup to wait for all workers to finish, and a results channel to collect output. Set the number of workers to runtime.NumCPU() for CPU-bound tasks, or higher for I/O-bound tasks.

How do I handle large JSONL files in Go?

Both bufio.Scanner and json.Decoder process records one at a time without loading the entire file. For Scanner, increase the buffer size with scanner.Buffer() if your lines exceed 64KB. For Decoder, it handles buffering internally. Combine either approach with goroutines for parallel processing of very large files.

Can I use json.Decoder with HTTP response bodies for JSONL?

Yes. json.Decoder works with any io.Reader, including http.Response.Body. Create a decoder with json.NewDecoder(resp.Body) and call Decode in a loop until you receive io.EOF. This is ideal for streaming JSONL from APIs because it processes records as they arrive over the network without waiting for the full response.

JSONL in Go: bufio, json.Decoder & Concurrency

A complete guide to working with JSONL (JSON Lines) files in Go. Learn to read, write, stream, and concurrently process JSONL data using the standard library's bufio, encoding/json, and goroutines.

Last updated: February 2026

Why Go for JSONL?

Go is an excellent choice for processing JSONL files, especially when performance and concurrency matter. The standard library provides everything you need: bufio for efficient line-by-line I/O, encoding/json for parsing and serialization, and goroutines for parallel processing. There are no third-party dependencies required. Go's compiled nature means your JSONL pipeline will run significantly faster than equivalent Python or Node.js scripts, often processing millions of lines per second on modest hardware.

JSONL (JSON Lines) stores one JSON object per line, making it a natural fit for Go's streaming I/O model. Instead of loading an entire file into memory, you can read and process records one at a time using a scanner or decoder. Combined with Go's lightweight goroutines and channels, you can build concurrent JSONL pipelines that saturate all available CPU cores. In this guide, you will learn how to read JSONL with bufio.Scanner, stream with json.Decoder, write JSONL efficiently, process records concurrently, and handle errors robustly.

Reading JSONL Files with bufio.Scanner

The most straightforward way to read JSONL in Go is with bufio.Scanner. It reads a file line by line, and you parse each line with json.Unmarshal. This approach gives you full control over buffer sizes and error handling.

Open the file, create a scanner, and iterate line by line. Each line is parsed into a map or struct with json.Unmarshal. This keeps memory usage proportional to a single record, not the entire file.

Basic Reading with bufio.Scanner

package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"log"
	"os"
)

func main() {
	file, err := os.Open("data.jsonl")
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	var records []map[string]any
	scanner := bufio.NewScanner(file)

	// Increase buffer for lines longer than 64KB
	scanner.Buffer(make([]byte, 0, 1024*1024), 1024*1024)

	for scanner.Scan() {
		var record map[string]any
		if err := json.Unmarshal(scanner.Bytes(), &record); err != nil {
			log.Printf("skipping invalid JSON: %v", err)
			continue
		}
		records = append(records, record)
	}

	if err := scanner.Err(); err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Loaded %d records\n", len(records))
}

For production code, parse JSONL records into typed Go structs instead of generic maps. This provides compile-time type safety, better performance, and clearer code. Define your struct with json tags to control field mapping.

Parsing into Typed Structs

package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"log"
	"os"
)

type User struct {
	ID    int    `json:"id"`
	Name  string `json:"name"`
	Email string `json:"email"`
	Age   int    `json:"age,omitempty"`
}

func readUsers(path string) ([]User, error) {
	file, err := os.Open(path)
	if err != nil {
		return nil, fmt.Errorf("open file: %w", err)
	}
	defer file.Close()

	var users []User
	scanner := bufio.NewScanner(file)
	lineNum := 0

	for scanner.Scan() {
		lineNum++
		var u User
		if err := json.Unmarshal(scanner.Bytes(), &u); err != nil {
			return nil, fmt.Errorf("line %d: %w", lineNum, err)
		}
		users = append(users, u)
	}

	if err := scanner.Err(); err != nil {
		return nil, fmt.Errorf("scanner error: %w", err)
	}
	return users, nil
}

func main() {
	users, err := readUsers("users.jsonl")
	if err != nil {
		log.Fatal(err)
	}
	for _, u := range users {
		fmt.Printf("%s (%s)\n", u.Name, u.Email)
	}
}

Streaming with json.Decoder

For high-performance streaming, json.Decoder reads directly from an io.Reader without intermediate line splitting. It handles buffering internally and is the recommended approach for large JSONL files or network streams where you want minimal allocations.

json.NewDecoder wraps any io.Reader and decodes JSON values sequentially. Each call to Decode reads exactly one JSON object. When the stream ends, it returns io.EOF. This is more efficient than bufio.Scanner + json.Unmarshal because it avoids copying each line into a separate buffer.

Streaming with json.Decoder

package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"log"
	"os"
)

type LogEntry struct {
	Timestamp string `json:"timestamp"`
	Level     string `json:"level"`
	Message   string `json:"message"`
	Service   string `json:"service"`
}

func processJSONLStream(path string) error {
	file, err := os.Open(path)
	if err != nil {
		return fmt.Errorf("open: %w", err)
	}
	defer file.Close()

	decoder := json.NewDecoder(file)
	var count int

	for {
		var entry LogEntry
		err := decoder.Decode(&entry)
		if errors.Is(err, io.EOF) {
			break
		}
		if err != nil {
			return fmt.Errorf("decode record %d: %w", count+1, err)
		}
		count++

		// Process each record as it's decoded
		if entry.Level == "ERROR" {
			fmt.Printf("[%s] %s: %s\n",
				entry.Timestamp, entry.Service, entry.Message)
		}
	}

	fmt.Printf("Processed %d log entries\n", count)
	return nil
}

func main() {
	if err := processJSONLStream("logs.jsonl"); err != nil {
		log.Fatal(err)
	}
}

Writing JSONL Files in Go

Writing JSONL in Go is straightforward: serialize each record with json.Marshal, append a newline byte, and write to a file. For best performance, wrap the file writer in a bufio.Writer to reduce system calls.

Use json.Marshal to serialize each record to JSON bytes, then write them followed by a newline. This approach is simple and works well for small to medium files.

Basic Writing with json.Marshal

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"os"
)

type Product struct {
	ID    int     `json:"id"`
	Name  string  `json:"name"`
	Price float64 `json:"price"`
}

func main() {
	products := []Product{
		{ID: 1, Name: "Laptop", Price: 999.99},
		{ID: 2, Name: "Mouse", Price: 29.99},
		{ID: 3, Name: "Keyboard", Price: 79.99},
	}

	file, err := os.Create("products.jsonl")
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	for _, p := range products {
		data, err := json.Marshal(p)
		if err != nil {
			log.Printf("skip marshal error: %v", err)
			continue
		}
		data = append(data, '\n')
		if _, err := file.Write(data); err != nil {
			log.Fatal(err)
		}
	}

	fmt.Printf("Wrote %d records to products.jsonl\n", len(products))
}

For high-throughput writing, use json.NewEncoder with a bufio.Writer. The encoder appends a newline after each Encode call automatically, and the buffered writer batches small writes into larger system calls. Always call Flush before closing the file.

Buffered Writing with json.Encoder

package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"log"
	"os"
)

type Event struct {
	Type      string         `json:"type"`
	Timestamp int64          `json:"timestamp"`
	Payload   map[string]any `json:"payload"`
}

func writeEvents(path string, events []Event) error {
	file, err := os.Create(path)
	if err != nil {
		return fmt.Errorf("create file: %w", err)
	}
	defer file.Close()

	writer := bufio.NewWriter(file)
	encoder := json.NewEncoder(writer)

	// Disable HTML escaping for cleaner output
	encoder.SetEscapeHTML(false)

	for i, event := range events {
		if err := encoder.Encode(event); err != nil {
			return fmt.Errorf("encode event %d: %w", i, err)
		}
	}

	// Flush buffered data to file
	if err := writer.Flush(); err != nil {
		return fmt.Errorf("flush: %w", err)
	}

	fmt.Printf("Wrote %d events to %s\n", len(events), path)
	return nil
}

func main() {
	events := []Event{
		{Type: "click", Timestamp: 1700000001, Payload: map[string]any{"x": 120, "y": 450}},
		{Type: "view", Timestamp: 1700000002, Payload: map[string]any{"page": "/home"}},
	}
	if err := writeEvents("events.jsonl", events); err != nil {
		log.Fatal(err)
	}
}

Concurrent JSONL Processing with Goroutines

Go's goroutines and channels make it easy to build concurrent JSONL pipelines. The fan-out pattern reads records from a file, distributes them across multiple worker goroutines, and collects results through a channel. This is ideal for CPU-bound transformations on large JSONL files.

Concurrent JSONL Processing with Goroutines

package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"log"
	"os"
	"runtime"
	"sync"
)

type Record struct {
	ID   int            `json:"id"`
	Data map[string]any `json:"data"`
}

type Result struct {
	ID        int  `json:"id"`
	Processed bool `json:"processed"`
	Score     int  `json:"score"`
}

func process(r Record) Result {
	// Simulate CPU-bound work
	score := len(r.Data) * r.ID
	return Result{ID: r.ID, Processed: true, Score: score}
}

func main() {
	file, err := os.Open("records.jsonl")
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	numWorkers := runtime.NumCPU()
	jobs := make(chan Record, numWorkers*2)
	results := make(chan Result, numWorkers*2)

	// Start workers
	var wg sync.WaitGroup
	for i := 0; i < numWorkers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for record := range jobs {
				results <- process(record)
			}
		}()
	}

	// Close results channel when all workers finish
	go func() {
		wg.Wait()
		close(results)
	}()

	// Read file and send records to workers
	go func() {
		scanner := bufio.NewScanner(file)
		for scanner.Scan() {
			var r Record
			if err := json.Unmarshal(scanner.Bytes(), &r); err != nil {
				log.Printf("skip invalid line: %v", err)
				continue
			}
			jobs <- r
		}
		close(jobs)
	}()

	// Collect and write results
	outFile, err := os.Create("results.jsonl")
	if err != nil {
		log.Fatal(err)
	}
	defer outFile.Close()

	writer := bufio.NewWriter(outFile)
	encoder := json.NewEncoder(writer)
	var count int

	for result := range results {
		if err := encoder.Encode(result); err != nil {
			log.Printf("encode error: %v", err)
		}
		count++
	}
	writer.Flush()

	fmt.Printf("Processed %d records with %d workers\n", count, numWorkers)
}

This fan-out pattern distributes JSONL records across goroutines equal to the number of CPU cores. The buffered channels prevent goroutines from blocking on send or receive. The sync.WaitGroup ensures all workers finish before the results channel is closed. For I/O-bound workloads such as API calls, you can increase numWorkers beyond the CPU count. Note that output order is not guaranteed with concurrent processing.

Error Handling Best Practices

Robust JSONL processing in Go requires careful error handling. Real-world JSONL files can contain malformed lines, encoding issues, or unexpectedly large records. This pattern provides line-level error tracking, configurable skip-on-error behavior, and summary reporting.

Error Handling Best Practices

package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"log"
	"os"
)

type ParseError struct {
	Line int
	Err  error
	Raw  string
}

func (e *ParseError) Error() string {
	return fmt.Sprintf("line %d: %v", e.Line, e.Err)
}

type JSONLReader struct {
	scanner  *bufio.Scanner
	lineNum  int
	Errors   []ParseError
	Skipped  int
	Success  int
}

func NewJSONLReader(file *os.File) *JSONLReader {
	scanner := bufio.NewScanner(file)
	scanner.Buffer(make([]byte, 0, 1024*1024), 10*1024*1024)
	return &JSONLReader{scanner: scanner}
}

func (r *JSONLReader) ReadAll(target any) error {
	slice, ok := target.(*[]map[string]any)
	if !ok {
		return fmt.Errorf("target must be *[]map[string]any")
	}

	for r.scanner.Scan() {
		r.lineNum++
		line := r.scanner.Bytes()

		// Skip empty lines
		if len(line) == 0 {
			continue
		}

		var record map[string]any
		if err := json.Unmarshal(line, &record); err != nil {
			r.Errors = append(r.Errors, ParseError{
				Line: r.lineNum,
				Err:  err,
				Raw:  string(line),
			})
			r.Skipped++
			continue
		}
		*slice = append(*slice, record)
		r.Success++
	}

	return r.scanner.Err()
}

func (r *JSONLReader) Summary() string {
	return fmt.Sprintf(
		"Lines: %d | Success: %d | Skipped: %d | Errors: %d",
		r.lineNum, r.Success, r.Skipped, len(r.Errors),
	)
}

func main() {
	file, err := os.Open("data.jsonl")
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	reader := NewJSONLReader(file)
	var records []map[string]any

	if err := reader.ReadAll(&records); err != nil {
		log.Fatal(err)
	}

	fmt.Println(reader.Summary())
	for _, e := range reader.Errors {
		log.Printf("Parse error at %s", e.Error())
	}
}

This JSONLReader struct tracks line numbers, collects parse errors with their raw content, and reports a summary of successes and failures. The 10MB scanner buffer handles unusually large lines without panicking. In production, you might extend this with context.Context support for cancellation, a maximum error threshold that aborts processing, or structured logging with slog.

Go Packages for JSONL Processing

Go's standard library provides all the building blocks you need for JSONL processing. Here are the three core approaches and when to use each one.

bufio.Scanner

Flexible

The standard line-by-line reader. Pairs with json.Unmarshal for explicit parsing. Best when you need control over buffer sizes, want to skip or inspect raw lines, or need line numbers for error reporting. Handles lines up to your configured buffer size.

json.Decoder

Recommended

Streaming JSON decoder that reads directly from an io.Reader. More efficient than Scanner + Unmarshal because it avoids copying data. Ideal for large files and network streams. Handles back-to-back JSON values automatically without needing explicit newline splitting.

json.Encoder

Writing

Streaming JSON encoder that writes to an io.Writer. Automatically appends a newline after each Encode call, making it perfect for JSONL output. Combine with bufio.Writer for high-throughput writing. Supports SetEscapeHTML and SetIndent configuration.

Try Our Free JSONL Tools

Don't want to write code? Use our free online tools to view, validate, and convert JSONL files right in your browser.

large JSONL files

JSONL best practices

JSONL validator

Work with JSONL Files Online

View, validate, and convert JSONL files up to 1GB right in your browser. No uploads required, 100% private.

JSONL in Go: bufio, json.Decoder & Concurrency

Why Go for JSONL?

Reading JSONL Files with bufio.Scanner

Streaming with json.Decoder

Writing JSONL Files in Go

Concurrent JSONL Processing with Goroutines

Error Handling Best Practices

Go Packages for JSONL Processing

bufio.Scanner

json.Decoder

json.Encoder

Try Our Free JSONL Tools

Work with JSONL Files Online

Frequently Asked Questions

How do I read a JSONL file in Go?

What is the difference between bufio.Scanner and json.Decoder for JSONL?

How do I write a JSONL file in Go?

How do I process JSONL concurrently in Go?

How do I handle large JSONL files in Go?

Can I use json.Decoder with HTTP response bodies for JSONL?