作为一个合格的图片加载框架,一般都会有内存缓存和硬盘缓存。在本篇,我们就先来看看 Glide 的硬盘缓存实现。

Glide 使用版本 4.7.1

接口说明

Glide 的硬盘缓存由接口 DiskCache 定义,用户可以根据需要,提供自己实现的 DiskCache,或者使用 Glide 内置的实现。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
/**
* An interface for writing to and reading from a disk cache.
*/
public interface DiskCache {

/**
* An interface for lazily creating a disk cache.
*/
interface Factory {
/** 250 MB of cache. */
int DEFAULT_DISK_CACHE_SIZE = 250 * 1024 * 1024;
String DEFAULT_DISK_CACHE_DIR = "image_manager_disk_cache";

/** Returns a new disk cache, or {@code null} if no disk cache could be created. */
@Nullable
DiskCache build();
}

/**
* An interface to actually write data to a key in the disk cache.
*/
interface Writer {
/**
* Writes data to the file and returns true if the write was successful and should be committed,
* and false if the write should be aborted.
*
* @param file The File the Writer should write to.
*/
boolean write(@NonNull File file);
}

/**
* Get the cache for the value at the given key.
*
* <p> Note - This is potentially dangerous, someone may write a new value to the file at any
* point in time and we won't know about it. </p>
*
* @param key The key in the cache.
* @return An InputStream representing the data at key at the time get is called.
*/
@Nullable
File get(Key key);

/**
* Write to a key in the cache. {@link Writer} is used so that the cache implementation can
* perform actions after the write finishes, like commit (via atomic file rename).
*
* @param key The key to write to.
* @param writer An interface that will write data given an OutputStream for the key.
*/
void put(Key key, Writer writer);

/**
* Remove the key and value from the cache.
*
* @param key The key to remove.
*/
// Public API.
@SuppressWarnings("unused")
void delete(Key key);

/**
* Clear the cache.
*/
void clear();
}

这个接口并没有太多值得玩味的地方,读者自己看一看就好。

Factory 工厂

DiskCache.Factory 作为一个工厂,用来 build 出实际的 DiskCache 实例。Glide 内部共有 3 个 Factory 的实现,分别是:

1
2
3
class DiskLruCacheFactory implements DiskCache.Factory;
class InternalCacheDiskCacheFactory extends DiskLruCacheFactory;
class ExternalPreferredCacheDiskCacheFactory extends DiskLruCacheFactory;

InternalCacheDiskCacheFactory 所创建的 DiskCache 将文件存储在 internal storage,存放在这里的文件只有我们自己能够读取。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
public final class InternalCacheDiskCacheFactory extends DiskLruCacheFactory {

public InternalCacheDiskCacheFactory(Context context) {
// 默认情况下,文件夹名字是 image_manager_disk_cache
// 缓存大小为 250M
this(context, DiskCache.Factory.DEFAULT_DISK_CACHE_DIR,
DiskCache.Factory.DEFAULT_DISK_CACHE_SIZE);
}

public InternalCacheDiskCacheFactory(Context context, long diskCacheSize) {
this(context, DiskCache.Factory.DEFAULT_DISK_CACHE_DIR, diskCacheSize);
}

public InternalCacheDiskCacheFactory(final Context context, final String diskCacheName,
long diskCacheSize) {
super(new CacheDirectoryGetter() {
@Override
public File getCacheDirectory() {
// 从这里可以看到,我们确实是用的 internal storage
File cacheDirectory = context.getCacheDir();
if (cacheDirectory == null) {
return null;
}
if (diskCacheName != null) {
return new File(cacheDirectory, diskCacheName);
}
return cacheDirectory;
}
}, diskCacheSize);
}
}

ExternalPreferredCacheDiskCacheFactory 优先使用 external storage,在 external storage 不可写的情况下,会转而使用 internal storage。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
/**
* Creates an {@link com.bumptech.glide.disklrucache.DiskLruCache} based disk cache in the external
* disk cache directory, which falls back to the internal disk cache if no external storage is
* available. If ever fell back to the internal disk cache, will use that one from that moment on.
*
* <p><b>Images can be read by everyone when using external disk cache.</b>
*/
// Public API.
@SuppressWarnings({"unused", "WeakerAccess"})
public final class ExternalPreferredCacheDiskCacheFactory extends DiskLruCacheFactory {

public ExternalPreferredCacheDiskCacheFactory(Context context) {
this(context, DiskCache.Factory.DEFAULT_DISK_CACHE_DIR,
DiskCache.Factory.DEFAULT_DISK_CACHE_SIZE);
}

public ExternalPreferredCacheDiskCacheFactory(Context context, long diskCacheSize) {
this(context, DiskCache.Factory.DEFAULT_DISK_CACHE_DIR, diskCacheSize);
}

public ExternalPreferredCacheDiskCacheFactory(final Context context, final String diskCacheName,
final long diskCacheSize) {
super(new CacheDirectoryGetter() {
@Nullable
private File getInternalCacheDirectory() {
File cacheDirectory = context.getCacheDir();
if (cacheDirectory == null) {
return null;
}
if (diskCacheName != null) {
return new File(cacheDirectory, diskCacheName);
}
return cacheDirectory;
}

@Override
public File getCacheDirectory() {
File internalCacheDirectory = getInternalCacheDirectory();

// Already used internal cache, so keep using that one,
// thus avoiding using both external and internal with transient errors.
if ((null != internalCacheDirectory) && internalCacheDirectory.exists()) {
return internalCacheDirectory;
}

File cacheDirectory = context.getExternalCacheDir();

// Shared storage is not available.
if ((cacheDirectory == null) || (!cacheDirectory.canWrite())) {
return internalCacheDirectory;
}
if (diskCacheName != null) {
return new File(cacheDirectory, diskCacheName);
}
return cacheDirectory;
}
}, diskCacheSize);
}
}

当使用 external storage 的时候,路径为 /sdcard/Android/data/your.package.name/,这里路径所有程序都可以访问。我们自己往里面写东西,也不需要声明权限,在应用被删除后,这个文件夹会被系统一起删除。

缓存不管是放在 internal storage 还是 external storage,唯一的差异,其实就是存放的路径不同。所以,InternalCacheDiskCacheFactoryExternalPreferredCacheDiskCacheFactory 都只是用于指定一个路径,实际的 factory 的职责由他们共同的父类 DiskLruCacheFactory 来完成。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
/**
* Creates an {@link com.bumptech.glide.disklrucache.DiskLruCache} based disk cache in the specified
* disk cache directory.
*
* <p>If you need to make I/O access before returning the cache directory use the {@link
* DiskLruCacheFactory#DiskLruCacheFactory(CacheDirectoryGetter, long)} constructor variant.
*/
// Public API.
@SuppressWarnings("unused")
public class DiskLruCacheFactory implements DiskCache.Factory {
private final long diskCacheSize;
private final CacheDirectoryGetter cacheDirectoryGetter;

/**
* Interface called out of UI thread to get the cache folder.
*/
public interface CacheDirectoryGetter {
File getCacheDirectory();
}

// 我们可以不使用 InternalCacheDiskCacheFactory、ExternalPreferredCacheDiskCacheFactory
// 直接通过这个接口构建 DiskLruCacheFactory。通过这种方式,我们可以指定任意的缓存路
public DiskLruCacheFactory(final String diskCacheFolder, long diskCacheSize) {
this(new CacheDirectoryGetter() {
@Override
public File getCacheDirectory() {
return new File(diskCacheFolder);
}
}, diskCacheSize);
}

public DiskLruCacheFactory(final String diskCacheFolder, final String diskCacheName,
long diskCacheSize) {
this(new CacheDirectoryGetter() {
@Override
public File getCacheDirectory() {
return new File(diskCacheFolder, diskCacheName);
}
}, diskCacheSize);
}

/**
* When using this constructor {@link CacheDirectoryGetter#getCacheDirectory()} will be called out
* of UI thread, allowing to do I/O access without performance impacts.
*
* @param cacheDirectoryGetter Interface called out of UI thread to get the cache folder.
* @param diskCacheSize Desired max bytes size for the LRU disk cache.
*/
// Public API.
@SuppressWarnings("WeakerAccess")
public DiskLruCacheFactory(CacheDirectoryGetter cacheDirectoryGetter, long diskCacheSize) {
this.diskCacheSize = diskCacheSize;
this.cacheDirectoryGetter = cacheDirectoryGetter;
}

@Override
public DiskCache build() {
File cacheDir = cacheDirectoryGetter.getCacheDirectory();

if (cacheDir == null) {
return null;
}

if (!cacheDir.mkdirs() && (!cacheDir.exists() || !cacheDir.isDirectory())) {
return null;
}

// 调用 DiskLruCacheWrapper 的工厂方法构造一个 DiskLruCacheWrapper,DiskLruCacheWrapper
// 是 glide 内建的 DiskCache 实现
return DiskLruCacheWrapper.create(cacheDir, diskCacheSize);
}
}

Factory 的实现很简单,到这里我们就看完了。下面我们看看 DiskLruCacheWrapper。

wrapper 实现

Glide 的硬盘缓存,实际上上并不是自己实现的,而是使用别人实现好的 disklrucache。在 build.gradle 可以看到:

1
2
3
4
5
6
7
// library/build.gradle

dependencies {
api project(':third_party:disklrucache')

// ...
}

由于 DiskCache 是 Glide 自动定义的接口,disklrucache 并没有实现它,所以这里我们借助 DiskLruCacheWrapper 来解决接口的问题(设计模式迷此时估计会大喊:adapter 模式!)。

DiskLruCacheWrapper 的构造

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
/**
* Create a new DiskCache in the given directory with a specified max size.
*
* @param directory The directory for the disk cache
* @param maxSize The max size for the disk cache
* @return The new disk cache with the given arguments
*/
@SuppressWarnings("deprecation")
public static DiskCache create(File directory, long maxSize) {
return new DiskLruCacheWrapper(directory, maxSize);
}

/**
* @deprecated Do not extend this class.
*/
@Deprecated
// Deprecated public API.
@SuppressWarnings({"WeakerAccess", "DeprecatedIsStillUsed"})
protected DiskLruCacheWrapper(File directory, long maxSize) {
this.directory = directory;
this.maxSize = maxSize;
this.safeKeyGenerator = new SafeKeyGenerator();
}

SafeKeyGenerator 用于将一个 Key 转换为 String 类型的 key,它跟主干逻辑没有太多关系,这里我们就不看了。

获取文件

获取文件由 get(Key) 完成:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
private DiskLruCache diskLruCache;


private synchronized DiskLruCache getDiskCache() throws IOException {
// 懒加载
// 注意,这里跟 double check 不一样,diskLruCache 并不需要是 volatile。具体原因这里就不展开谈了。
if (diskLruCache == null) {
diskLruCache = DiskLruCache.open(directory, APP_VERSION, VALUE_COUNT, maxSize);
}
return diskLruCache;
}

@Override
public File get(Key key) {
String safeKey = safeKeyGenerator.getSafeKey(key);
if (Log.isLoggable(TAG, Log.VERBOSE)) {
Log.v(TAG, "Get: Obtained: " + safeKey + " for for Key: " + key);
}
File result = null;
try {
// It is possible that the there will be a put in between these two gets. If so that shouldn't
// be a problem because we will always put the same value at the same key so our input streams
// will still represent the same data.
final DiskLruCache.Value value = getDiskCache().get(safeKey);
if (value != null) {
result = value.getFile(0);
}
} catch (IOException e) {
if (Log.isLoggable(TAG, Log.WARN)) {
Log.w(TAG, "Unable to get from disk cache", e);
}
}
return result;
}

存储文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
private final DiskCacheWriteLocker writeLocker = new DiskCacheWriteLocker();


@Override
public void put(Key key, Writer writer) {
// We want to make sure that puts block so that data is available when put completes. We may
// actually not write any data if we find that data is written by the time we acquire the lock.
String safeKey = safeKeyGenerator.getSafeKey(key);
// 获取针对 safeKey 的锁
writeLocker.acquire(safeKey);
try {
if (Log.isLoggable(TAG, Log.VERBOSE)) {
Log.v(TAG, "Put: Obtained: " + safeKey + " for for Key: " + key);
}
try {
// We assume we only need to put once, so if data was written while we were trying to get
// the lock, we can simply abort.
DiskLruCache diskCache = getDiskCache();
Value current = diskCache.get(safeKey);
if (current != null) {
return;
}

DiskLruCache.Editor editor = diskCache.edit(safeKey);
if (editor == null) {
throw new IllegalStateException("Had two simultaneous puts for: " + safeKey);
}
try {
File file = editor.getFile(0);
if (writer.write(file)) {
editor.commit();
}
} finally {
// DiskLruCache 实现了日志系统,写文件作为一个事务来处理
editor.abortUnlessCommitted();
}
} catch (IOException e) {
if (Log.isLoggable(TAG, Log.WARN)) {
Log.w(TAG, "Unable to put to disk cache", e);
}
}
} finally {
writeLocker.release(safeKey);
}
}

写文件的时候,不同的文件直接的写并不会相互影响,writeLocker.acquire(safeKey) 中会给每个 key 分配一个锁。关于 DiskCacheWriteLocker 的实现,由于篇幅关系,这里就不看了。

删除文件、情况缓存

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
@Override
public void delete(Key key) {
String safeKey = safeKeyGenerator.getSafeKey(key);
try {
getDiskCache().remove(safeKey);
} catch (IOException e) {
if (Log.isLoggable(TAG, Log.WARN)) {
Log.w(TAG, "Unable to delete from disk cache", e);
}
}
}

@Override
public synchronized void clear() {
try {
getDiskCache().delete();
} catch (IOException e) {
if (Log.isLoggable(TAG, Log.WARN)) {
Log.w(TAG, "Unable to clear disk cache or disk cache cleared externally", e);
}
} finally {
// Delete can close the cache but still throw. If we don't null out the disk cache here, every
// subsequent request will try to act on a closed disk cache and fail. By nulling out the disk
// cache we at least allow for attempts to open the cache in the future. See #2465.
resetDiskCache();
}
}

private synchronized void resetDiskCache() {
diskLruCache = null;
}

这两个方法是在没有什么值得说的。下面我们接着看 DiskLruCache 的实现。

DiskLruCache 实现

在看代码之前,我们先来聊聊为什么需要日志。最简单的情况,我们想要把某个图片缓存起来,由于写硬盘是一个比较耗时的操作,我们把它放到一个后台线程来执行。假设在写文件的过程中,很不幸的,其他某个线程抛出了未捕获的异常,导致程序终止了。这时候,我们刚才想要保存的图片,其实才写了一半。在这种情况下,我们就需要一种机制,可以表明这个文件是损坏了的,在程序下次启动的时候移它。

为了实现日志(journal)系统,一般的思路是,比方说上面的情况,我们在保存文件的前,先往一个日志文件写入一条日志,说我们马上就要写某某某文件了。写完后,我们再来执行文件保存动作。保存成功后,我们再一次修改日志文件,说我们已经保存成功了。

这时候,万一我们在保存文件的过程中跪了,程序下次启动的时候,只要检查日志文件,就知道哪些文件是损坏的(最倒霉的情况是,我们保存成功,在修改日志的时候跪了。此时,正常的这个文件也会被清除,但是,对于一个缓存,这又有什么所谓呢)。

我们先看看 DiskLruCache 里面关于日志文件的一段注释,它对我们理解代码有非常大的帮助:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
/*
* This cache uses a journal file named "journal". A typical journal file
* looks like this:
* libcore.io.DiskLruCache
* 1
* 100
* 2
*
* CLEAN 3400330d1dfc7f3f7f4b8d4d803dfcf6 832 21054
* DIRTY 335c4c6028171cfddfbaae1a9c313c52
* CLEAN 335c4c6028171cfddfbaae1a9c313c52 3934 2342
* REMOVE 335c4c6028171cfddfbaae1a9c313c52
* DIRTY 1ab96a171faeeee38496d8b330771a7a
* CLEAN 1ab96a171faeeee38496d8b330771a7a 1600 234
* READ 335c4c6028171cfddfbaae1a9c313c52
* READ 3400330d1dfc7f3f7f4b8d4d803dfcf6
*
* The first five lines of the journal form its header. They are the
* constant string "libcore.io.DiskLruCache", the disk cache's version,
* the application's version, the value count, and a blank line.
*
* Each of the subsequent lines in the file is a record of the state of a
* cache entry. Each line contains space-separated values: a state, a key,
* and optional state-specific values.
* o DIRTY lines track that an entry is actively being created or updated.
* Every successful DIRTY action should be followed by a CLEAN or REMOVE
* action. DIRTY lines without a matching CLEAN or REMOVE indicate that
* temporary files may need to be deleted.
* o CLEAN lines track a cache entry that has been successfully published
* and may be read. A publish line is followed by the lengths of each of
* its values.
* o READ lines track accesses for LRU.
* o REMOVE lines track entries that have been deleted.
*
* The journal file is appended to as cache operations occur. The journal may
* occasionally be compacted by dropping redundant lines. A temporary file named
* "journal.tmp" will be used during compaction; that file should be deleted if
* it exists when the cache is opened.
*/

DiskLruCache 实例的创建

上面看 DiskLruCacheWrapper 的时候我们就知道,创建 DiskLruCache 时使用的是它的静态方法 open

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
/**
* Opens the cache in {@code directory}, creating a cache if none exists
* there.
*
* @param directory a writable directory
* @param valueCount the number of values per cache entry. Must be positive.
* @param maxSize the maximum number of bytes this cache should use to store
* @throws IOException if reading or writing the cache directory fails
*/
public static DiskLruCache open(File directory, int appVersion, int valueCount, long maxSize)
throws IOException {
if (maxSize <= 0) {
throw new IllegalArgumentException("maxSize <= 0");
}
if (valueCount <= 0) {
throw new IllegalArgumentException("valueCount <= 0");
}

// If a bkp file exists, use it instead.
File backupFile = new File(directory, JOURNAL_FILE_BACKUP);
if (backupFile.exists()) {
File journalFile = new File(directory, JOURNAL_FILE);
// If journal file also exists just delete backup file.
if (journalFile.exists()) {
backupFile.delete();
} else {
renameTo(backupFile, journalFile, false);
}
}

// Prefer to pick up where we left off.
DiskLruCache cache = new DiskLruCache(directory, appVersion, valueCount, maxSize);
if (cache.journalFile.exists()) {
try {
cache.readJournal();
cache.processJournal();
return cache;
} catch (IOException journalIsCorrupt) {
System.out
.println("DiskLruCache "
+ directory
+ " is corrupt: "
+ journalIsCorrupt.getMessage()
+ ", removing");
cache.delete();
}
}

// Create a new empty cache.
directory.mkdirs();
cache = new DiskLruCache(directory, appVersion, valueCount, maxSize);
cache.rebuildJournal();
return cache;
}

可以看到,跟我们上面讲的差不多,在创建的时候,会读取日志文件,然后做一些清理工作。下面我们先看 readJournal 方法:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
private void readJournal() throws IOException {
StrictLineReader reader = new StrictLineReader(new FileInputStream(journalFile), Util.US_ASCII);
try {
String magic = reader.readLine();
String version = reader.readLine();
String appVersionString = reader.readLine();
String valueCountString = reader.readLine();
String blank = reader.readLine();
// 判断一些文件头。关于文件头的格式,在上面那段注释中有说明
if (!MAGIC.equals(magic)
|| !VERSION_1.equals(version)
|| !Integer.toString(appVersion).equals(appVersionString)
|| !Integer.toString(valueCount).equals(valueCountString)
|| !"".equals(blank)) {
throw new IOException("unexpected journal header: [" + magic + ", " + version + ", "
+ valueCountString + ", " + blank + "]");
}

int lineCount = 0;
while (true) {
try {
// 读取并处理一条日志记录
readJournalLine(reader.readLine());
lineCount++;
} catch (EOFException endOfJournal) {
break;
}
}
// readJournalLine 会把已经 REMOVE 的 entry 移除掉,多出来的行数是冗余的(redundant);
// 并且,一个缓存项也可能对应多行(DIRTY/CLEAN/READ/...),但其实只需要一个 CLEAN 就够了。
// 当冗余的行数过多的时候,我们就需要清理日志文件
redundantOpCount = lineCount - lruEntries.size();

// 写了一半,应用就跪了
// If we ended on a truncated line, rebuild the journal before appending to it.
if (reader.hasUnterminatedLine()) {
rebuildJournal();
} else {
journalWriter = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(journalFile, true), Util.US_ASCII));
}
} finally {
Util.closeQuietly(reader);
}
}

private void readJournalLine(String line) throws IOException {
int firstSpace = line.indexOf(' ');
if (firstSpace == -1) {
throw new IOException("unexpected journal line: " + line);
}

int keyBegin = firstSpace + 1;
int secondSpace = line.indexOf(' ', keyBegin);
final String key;
if (secondSpace == -1) {
key = line.substring(keyBegin);
if (firstSpace == REMOVE.length() && line.startsWith(REMOVE)) {
// 每个操作就会记录一个日志信息,当我们删除一个缓存项的时候,添加一个 REMOVE。
// 既然缓存都已经 remove 掉了,这个 entry 也就就没有存在的必要的了
// 在 REMOVE 前,一定会有对应的 DIRTY/CLEAN 等,所以此时这个 key 对应的 entry
// 一定是存在的
lruEntries.remove(key);
return;
}
} else {
key = line.substring(keyBegin, secondSpace);
}

// 一个 Entry 代表缓存中的一项
Entry entry = lruEntries.get(key);
if (entry == null) {
entry = new Entry(key);
lruEntries.put(key, entry);
}

if (secondSpace != -1 && firstSpace == CLEAN.length() && line.startsWith(CLEAN)) {
String[] parts = line.substring(secondSpace + 1).split(" ");
entry.readable = true;
entry.currentEditor = null;
entry.setLengths(parts);
} else if (secondSpace == -1 && firstSpace == DIRTY.length() && line.startsWith(DIRTY)) {
entry.currentEditor = new Editor(entry);
} else if (secondSpace == -1 && firstSpace == READ.length() && line.startsWith(READ)) {
// This work was already done by calling lruEntries.get().
} else {
throw new IOException("unexpected journal line: " + line);
}
}

下面我们再看看 rebuildJournal

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
/**
* Creates a new journal that omits redundant information. This replaces the
* current journal if it exists.
*/
private synchronized void rebuildJournal() throws IOException {
if (journalWriter != null) {
journalWriter.close();
}

Writer writer = new BufferedWriter(
// 注意,这里我们写的是 tmp 文件
new OutputStreamWriter(new FileOutputStream(journalFileTmp), Util.US_ASCII));
try {
writer.write(MAGIC);
writer.write("\n");
writer.write(VERSION_1);
writer.write("\n");
writer.write(Integer.toString(appVersion));
writer.write("\n");
writer.write(Integer.toString(valueCount));
writer.write("\n");
writer.write("\n");

for (Entry entry : lruEntries.values()) {
if (entry.currentEditor != null) {
writer.write(DIRTY + ' ' + entry.key + '\n');
} else {
writer.write(CLEAN + ' ' + entry.key + entry.getLengths() + '\n');
}
}
} finally {
writer.close();
}

if (journalFile.exists()) {
renameTo(journalFile, journalFileBackup, true);
}
renameTo(journalFileTmp, journalFile, false);
journalFileBackup.delete();

journalWriter = new BufferedWriter(
new OutputStreamWriter(new FileOutputStream(journalFile, true), Util.US_ASCII));
}

private static void renameTo(File from, File to, boolean deleteDestination) throws IOException {
if (deleteDestination) {
deleteIfExists(to);
}
if (!from.renameTo(to)) {
throw new IOException();
}
}

这个方法很简单,就是先写一个文件头,然后把已有的 entry 一条一条写进去。

比较有意思的是后面那几个 rename。前面我们说,日志文件是为了在程序崩溃(还有内核崩溃,但这个比较少见)时恢复信息用的。当日志中冗余的行数太多的时候,我们也需要 rebuild 一下。在这里,我们同样需要写文件,万一在 rebuild journal 的时候跪了呢?!

解决办法就是,原来的日志文件不动,我们先把新的日志写到一个 tmp 文件里,等写完再用一个相对清理的 rename 来替换文件。由于 rename 的时候会删除原文件,这就存在一种可能,我们删除了 journal,而 rename tmp to journal 却失败了。此时,日志就丢失了。

所以,在把 tmp rename 为 journal 前,我们又把 journal rename 为 backup。在这种情况下,如果第一个 rename 成功,第二个失败,我们就能够通过这个 backup 来恢复原来的日志(在 DiskLruCache 的构造函数中,我们就检查了 backup 是否存在)。接着,如果第二个 rename 成功了,那么 backup 就没用了,于是删掉它。

Entry 的主要代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
private final class Entry {
private final String key;

/** Lengths of this entry's files. */
private final long[] lengths;

/** Memoized File objects for this entry to avoid char[] allocations. */
File[] cleanFiles;
File[] dirtyFiles;

/** True if this entry has ever been published. */
private boolean readable;

/** The ongoing edit or null if this entry is not being edited. */
private Editor currentEditor;

/** The sequence number of the most recently committed edit to this entry. */
private long sequenceNumber;

private Entry(String key) {
this.key = key;
this.lengths = new long[valueCount];
cleanFiles = new File[valueCount];
dirtyFiles = new File[valueCount];

// The names are repetitive so re-use the same builder to avoid allocations.
StringBuilder fileBuilder = new StringBuilder(key).append('.');
int truncateTo = fileBuilder.length();
for (int i = 0; i < valueCount; i++) {
fileBuilder.append(i);
cleanFiles[i] = new File(directory, fileBuilder.toString());
fileBuilder.append(".tmp");
dirtyFiles[i] = new File(directory, fileBuilder.toString());
fileBuilder.setLength(truncateTo);
}
}
}

可以看到,对于 entry 中的每个数据项,都有对应 dirty/clean 两个文件。这个也是为了数据的一致性而存在的,后面我们会看到他们的作用。

读取完日志文件后,DiskLruCache 又调用 processJournal 来处理它:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
/**
* Computes the initial size and collects garbage as a part of opening the
* cache. Dirty entries are assumed to be inconsistent and will be deleted.
*/
private void processJournal() throws IOException {
deleteIfExists(journalFileTmp);
for (Iterator<Entry> i = lruEntries.values().iterator(); i.hasNext(); ) {
Entry entry = i.next();
if (entry.currentEditor == null) { // 这个 entry 处于 CLEAN 状态
for (int t = 0; t < valueCount; t++) {
// size 是当前硬盘缓存占用的字节数
size += entry.lengths[t];
}
} else {
// DIRTY,说明上次有人写了一半就跪了。这种情况下,需要删除对应的文件
entry.currentEditor = null;
// valueCount 是我们在构造 DiskLruCache 时传递进来的,表示一个 entry 有多少个 value
// 我们用它来缓存文件,一个 entry 对应一个文件,valueCount == 1
for (int t = 0; t < valueCount; t++) {
deleteIfExists(entry.getCleanFile(t));
deleteIfExists(entry.getDirtyFile(t));
}
i.remove();
}
}
}

到这里,DiskLruCache 就构建完成了。

写文件到缓存

我们先来回顾一下 DiskLruCacheWrapper 中是怎么写文件的:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public void put(Key key, Writer writer) {
// ...

DiskLruCache.Editor editor = diskCache.edit(safeKey);
if (editor == null) {
throw new IllegalStateException("Had two simultaneous puts for: " + safeKey);
}
try {
File file = editor.getFile(0);
if (writer.write(file)) {
editor.commit();
}
} finally {
editor.abortUnlessCommitted();
}

// ...
}

基本上分为 4 步:

  1. 获取一个 editor
  2. 写文件
  3. commit
  4. 如果失败了,做一些清理工作

下面我们一个个来看。

  1. 获取一个 editor
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
/**
* Returns an editor for the entry named {@code key}, or null if another
* edit is in progress.
*/
public Editor edit(String key) throws IOException {
return edit(key, ANY_SEQUENCE_NUMBER);
}

private synchronized Editor edit(String key, long expectedSequenceNumber) throws IOException {
checkNotClosed();
Entry entry = lruEntries.get(key);
if (expectedSequenceNumber != ANY_SEQUENCE_NUMBER && (entry == null
|| entry.sequenceNumber != expectedSequenceNumber)) {
return null; // Value is stale.
}
if (entry == null) {
// 新建一个缓存
entry = new Entry(key);
lruEntries.put(key, entry);
} else if (entry.currentEditor != null) {
return null; // Another edit is in progress.
}

Editor editor = new Editor(entry);
entry.currentEditor = editor;

// Flush the journal before creating files to prevent file leaks.
journalWriter.append(DIRTY);
journalWriter.append(' ');
journalWriter.append(key);
journalWriter.append('\n');
journalWriter.flush();
return editor;
}

这里验证了我上面说的,在实际写文件前,先写了一个日志(啊,当然,你也可以说我是先看了代码,才写了上面那段话。如果如果能够做到这个,也是很不错的。学习一个特定的知识点,把它泛化后,你就得到了一个通用的问题解决方案)。

  1. 写文件
1
2
File file = editor.getFile(0);
writer.write(file);

写文件分两步,或者一个 file,然后 write 进去实际的数据。前面我们提到过,valueCount 代表每个缓存项中的数据数,我们只是存一个文件,所以这个 getFile(0) 获取第一个数据项(也是唯一的一个)。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
public File getFile(int index) throws IOException {
synchronized (DiskLruCache.this) {
if (entry.currentEditor != this) {
throw new IllegalStateException();
}
if (!entry.readable) {
written[index] = true;
}
File dirtyFile = entry.getDirtyFile(index);
if (!directory.exists()) {
directory.mkdirs();
}
return dirtyFile;
}
}

这里我们知道,写入的实际上是 dirty file。

  1. commit
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
/**
* Commits this edit so it is visible to readers. This releases the
* edit lock so another edit may be started on the same key.
*/
public void commit() throws IOException {
// The object using this Editor must catch and handle any errors
// during the write. If there is an error and they call commit
// anyway, we will assume whatever they managed to write was valid.
// Normally they should call abort.
completeEdit(this, true);
committed = true;
}


private synchronized void completeEdit(Editor editor, boolean success) throws IOException {
Entry entry = editor.entry;
if (entry.currentEditor != editor) {
throw new IllegalStateException();
}

// If this edit is creating the entry for the first time, every index must have a value.
if (success && !entry.readable) {
for (int i = 0; i < valueCount; i++) {
if (!editor.written[i]) {
editor.abort();
throw new IllegalStateException("Newly created entry didn't create value for index " + i);
}
if (!entry.getDirtyFile(i).exists()) {
editor.abort();
return;
}
}
}

for (int i = 0; i < valueCount; i++) {
File dirty = entry.getDirtyFile(i);
if (success) {
if (dirty.exists()) {
File clean = entry.getCleanFile(i);
dirty.renameTo(clean);
long oldLength = entry.lengths[i];
long newLength = clean.length();
entry.lengths[i] = newLength;
size = size - oldLength + newLength;
}
} else {
deleteIfExists(dirty);
}
}

redundantOpCount++;
entry.currentEditor = null;
if (entry.readable | success) {
entry.readable = true;
journalWriter.append(CLEAN);
journalWriter.append(' ');
journalWriter.append(entry.key);
journalWriter.append(entry.getLengths());
journalWriter.append('\n');

if (success) {
entry.sequenceNumber = nextSequenceNumber++;
}
} else {
// 写失败,对应的缓存项也不再有效
lruEntries.remove(entry.key);
journalWriter.append(REMOVE);
journalWriter.append(' ');
journalWriter.append(entry.key);
journalWriter.append('\n');
}
journalWriter.flush();

// 使用的总空间太多,或者冗余的日志太多
if (size > maxSize || journalRebuildRequired()) {
executorService.submit(cleanupCallable);
}
}


private final Callable<Void> cleanupCallable = new Callable<Void>() {
public Void call() throws Exception {
synchronized (DiskLruCache.this) {
if (journalWriter == null) {
return null; // Closed.
}
trimToSize();
if (journalRebuildRequired()) {
rebuildJournal();
redundantOpCount = 0;
}
}
return null;
}
};

private void trimToSize() throws IOException {
while (size > maxSize) {
Map.Entry<String, Entry> toEvict = lruEntries.entrySet().iterator().next();
remove(toEvict.getKey());
}
}

这里唯一需要注意的是,我们必须在 rename dirty to clean 成功后,才能写一个 CLEAN 日志。否则,如果我们写入 CLEAN 成功但 rename 却失败了,就会导致客户读取到错误的缓存数据。

  1. 如果失败了,做一些清理工作
1
editor.abortUnlessCommitted();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/**
* Aborts this edit. This releases the edit lock so another edit may be
* started on the same key.
*/
public void abort() throws IOException {
// 传入 false 会导致对应的 entry 被删除
completeEdit(this, false);
}

public void abortUnlessCommitted() {
if (!committed) {
try {
abort();
} catch (IOException ignored) {
}
}
}

下面,我们再看读文件部分。

从缓存中读取文件

还是一样,我们先看看 DiskLruCacheWrapper 中对 DiskLruCache 的调用方式:

1
2
3
4
5
6
7
8
9
10
public File get(Key key) {
// ...

final DiskLruCache.Value value = getDiskCache().get(safeKey);
if (value != null) {
result = value.getFile(0);
}

// ...
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
/**
* Returns a snapshot of the entry named {@code key}, or null if it doesn't
* exist is not currently readable. If a value is returned, it is moved to
* the head of the LRU queue.
*/
public synchronized Value get(String key) throws IOException {
checkNotClosed();
Entry entry = lruEntries.get(key);
if (entry == null) {
return null;
}

if (!entry.readable) {
return null;
}

for (File file : entry.cleanFiles) {
// A file must have been deleted manually!
if (!file.exists()) {
return null;
}
}

redundantOpCount++;
journalWriter.append(READ);
journalWriter.append(' ');
journalWriter.append(key);
journalWriter.append('\n');
if (journalRebuildRequired()) {
executorService.submit(cleanupCallable);
}

return new Value(key, entry.sequenceNumber, entry.cleanFiles, entry.lengths);
}

这个方法比较简单,读者自己看一看就好。

删除一个缓存项

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
/**
* Drops the entry for {@code key} if it exists and can be removed. Entries
* actively being edited cannot be removed.
*
* @return true if an entry was removed.
*/
public synchronized boolean remove(String key) throws IOException {
checkNotClosed();
Entry entry = lruEntries.get(key);
if (entry == null || entry.currentEditor != null) {
return false;
}

for (int i = 0; i < valueCount; i++) {
File file = entry.getCleanFile(i);
if (file.exists() && !file.delete()) {
throw new IOException("failed to delete " + file);
}
size -= entry.lengths[i];
entry.lengths[i] = 0;
}

redundantOpCount++;
journalWriter.append(REMOVE);
journalWriter.append(' ');
journalWriter.append(key);
journalWriter.append('\n');

lruEntries.remove(key);

if (journalRebuildRequired()) {
executorService.submit(cleanupCallable);
}

return true;
}

和前面其他文件况类似,这里我们需要先删除文件,然后再写日志。这样一来,即便文件删除成功而写日志失败,只要在下次读取时检测文件是否存在就可以了。如果反过来写日志成功但删除文件失败,则会导致缓存文件残留下来。

清空缓存

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
/**
* Closes the cache and deletes all of its stored values. This will delete
* all files in the cache directory including files that weren't created by
* the cache.
*/
public void delete() throws IOException {
close();
// 所有的缓存文件都放在 directory 里,整个目录删掉就“清空”了
Util.deleteContents(directory);
}

/** Closes this cache. Stored values will remain on the filesystem. */
public synchronized void close() throws IOException {
if (journalWriter == null) {
return; // Already closed.
}
for (Entry entry : new ArrayList<Entry>(lruEntries.values())) {
if (entry.currentEditor != null) {
entry.currentEditor.abort();
}
}
trimToSize();
journalWriter.close();
journalWriter = null;
}

不容易啊,到这里,DiskCache 就算是讲完了。


2018.06.20 补几个面试遇到的问题

日志文件的作用

  1. 应用崩溃后回收损坏的文件
  2. 日志文件记录了访问的顺序,通过日志文件,我们可以重新建立 LRU

考虑日志文件对回收损坏文件的作用,不用日志文件能够解决吗?

不能,因为我们没有办法知道一个文件是否是有效文件。