网站访问流程在线之家

今天看到一段源码 https://github.com/google-research/google-research/blob/master/instruction_following_eval/instructions_util.py 如下，对其中使用的装饰器函数感到好奇，所以产生了这篇博客。

@functools.lru_cache(maxsize=None)
def _get_sentence_tokenizer():return nltk.data.load("nltk:tokenizers/punkt/english.pickle")

Python中的@functools.lru_cache：用法、来源与应用

在Python中，@是装饰器语法的标志，用于将一个函数包装进另一个函数中，扩展或修改其功能。functools.lru_cache是Python标准库functools模块提供的一个装饰器，用于缓存函数的返回值，提升性能。

---

1. 装饰器的来源与语法

装饰器（Decorators）是Python函数式编程的一个核心特性。其来源可以追溯到Python 2.4版本，主要目的是让代码更简洁、更易读。@符号将装饰器函数与目标函数绑定，简化代码调用。

• 装饰器定义
  本质上，装饰器是一个接收函数作为参数并返回一个新函数的高阶函数。

示例：

def decorator(func):def wrapper():print("装饰器已执行")return func()return wrapper@decorator  # 等价于 func = decorator(func)
def say_hello():print("Hello, World!")

更多关于python装饰器的内容，可以参考笔者的另一篇博客：什么是装饰器？以Python为例：Basic Usage of Decorators （中英双语）

---

2. functools.lru_cache的作用

functools.lru_cache 是一个用于缓存函数调用结果的装饰器。

• LRU全称是Least Recently Used，即最近最少使用缓存策略。
• 当函数被多次调用时，lru_cache会缓存函数的输入和输出，减少重复计算，提升效率。

参数说明：

• maxsize: 设置缓存大小（None表示无限制）。
• typed: 是否区分输入参数的类型（默认为False）。

示例：

import functools@functools.lru_cache(maxsize=3)
def fibonacci(n):print(f"计算Fibonacci({n})")if n < 2:return nreturn fibonacci(n-1) + fibonacci(n-2)print(fibonacci(5))  # 计算结果缓存，重复调用时不会重新计算

输出示例：

计算Fibonacci(5)
计算Fibonacci(4)
计算Fibonacci(3)
计算Fibonacci(2)
计算Fibonacci(1)
计算Fibonacci(0)
5

---

3. 代码分析与解释

以下是https://github.com/google-research/google-research/blob/master/instruction_following_eval/instructions_util.py 提供的代码：

import functools
import nltkdef count_words(text):"""Counts the number of words."""tokenizer = nltk.tokenize.RegexpTokenizer(r"\w+")tokens = tokenizer.tokenize(text)num_words = len(tokens)return num_words@functools.lru_cache(maxsize=None)
def _get_sentence_tokenizer():return nltk.data.load("nltk:tokenizers/punkt/english.pickle")

代码解释：

1. count_words函数

   • 用于统计文本中的单词数量。
   • 通过nltk库中的RegexpTokenizer按正则表达式匹配单词，返回token列表并计算长度。

2. _get_sentence_tokenizer函数

   • 加载NLTK库的句子分割模型punkt。
   • 使用@functools.lru_cache(maxsize=None)装饰，表示缓存该函数的返回值。
   • 好处：加载分词器的操作是耗时的，通过缓存，只需加载一次。

---

4. functools.lru_cache的应用场景

• 递归函数：如斐波那契数列，避免重复计算。
• I/O密集型函数：比如数据库查询、网络请求，缓存结果以减少耗时。
• 大规模数据处理：缓存部分中间结果，提高性能。
• 机器学习/自然语言处理：模型加载、分词器初始化等耗时操作。

示例：

@functools.lru_cache(maxsize=128)
def fetch_data_from_db(query):# 模拟数据库查询print("执行查询:", query)return f"结果_{query}"print(fetch_data_from_db("SELECT * FROM table"))
print(fetch_data_from_db("SELECT * FROM table"))  # 直接返回缓存结果

输出：

执行查询: SELECT * FROM table
结果_SELECT * FROM table
结果_SELECT * FROM table

---

5. 补充内容：查看缓存信息

functools.lru_cache 提供了缓存管理的方法：

• cache_info()：查看缓存命中率和状态。
• cache_clear()：清空缓存。

示例：

@functools.lru_cache(maxsize=2)
def add(a, b):return a + badd(1, 2)
add(3, 4)
add(5, 6)print(add.cache_info())  # 输出缓存状态
add.cache_clear()        # 清除缓存

---

总结

1. 装饰器的语法：@用来将装饰器与函数绑定。
2. lru_cache作用：缓存函数结果，提升性能，避免重复计算。
3. 实际应用：适用于递归计算、I/O密集型操作和数据处理任务。
4. 代码示例：通过数值分析，展示了lru_cache的高效性。

通过合理使用functools.lru_cache，Python程序可以在性能瓶颈处得到极大提升。

英文版

Exploring @functools.lru_cache in Python

The @ symbol in Python is used for decorators, which allow you to modify or enhance the behavior of a function or method. One such decorator is functools.lru_cache, which caches a function’s return values for optimized performance, especially in repetitive or resource-intensive tasks.

---

What is lru_cache?

functools.lru_cache is part of Python’s functools module. LRU stands for Least Recently Used, a caching strategy that keeps only the most frequently or recently used results, removing older ones when the cache exceeds its size limit.

Key Syntax:

@functools.lru_cache(maxsize=None, typed=False)

Parameters:

1. maxsize: The maximum number of results to cache. If set to None, the cache size is unlimited.
2. typed: If True, different types of arguments (e.g., 1 vs. 1.0) will have separate cached results.

---

Code Example and Explanation

Here is an example of lru_cache in action: Source: https://github.com/google-research/google-research/blob/master/instruction_following_eval/instructions_util.py

import functools
import nltkdef count_words(text):"""Counts the number of words."""tokenizer = nltk.tokenize.RegexpTokenizer(r"\w+")tokens = tokenizer.tokenize(text)num_words = len(tokens)return num_words@functools.lru_cache(maxsize=None)
def _get_sentence_tokenizer():return nltk.data.load("nltk:tokenizers/punkt/english.pickle")

Breakdown:

1. count_words:

   • Uses RegexpTokenizer from NLTK to split a text into tokens (words).
   • Computes and returns the word count.

2. _get_sentence_tokenizer:

   • Loads NLTK’s pre-trained sentence tokenizer. This is resource-heavy if loaded multiple times.
   • Why use @lru_cache?:
     • With lru_cache, the tokenizer is loaded only once. Future calls retrieve it from the cache, saving time and memory.

---

How lru_cache Works

Internally, lru_cache uses a dictionary-like data structure to store results. When a function is called:

1. The function’s arguments act as a unique key.
2. If the result exists in the cache (cache hit), it is returned directly.
3. If not (cache miss), the function computes the result, stores it in the cache, and returns it.

---

Practical Applications

1. Recursive Calculations:

   @functools.lru_cache(maxsize=128)
   def fibonacci(n):if n < 2:return nreturn fibonacci(n - 1) + fibonacci(n - 2)
   

   In this example, repeated calls to the same input are avoided by caching results.

2. Optimizing Expensive Operations: Tasks like API calls, database queries, or loading large models can benefit greatly from caching.

---

Why Use lru_cache?

1. Performance Boost: Speeds up applications by avoiding redundant computations.
2. Memory Efficiency: Automatically clears old entries when the cache limit is reached.
3. Flexibility: It works with various types of functions and arguments.

---

Summary

@functools.lru_cache is a powerful tool for improving Python application efficiency. By caching results of expensive or repetitive function calls, it saves time and computational resources. Its ease of use and wide applicability make it an essential feature for developers.

后记

2024年12月16日19点49分于上海，在GPT4o大模型辅助下完成。